HIP-0107: Streaming Replication over VFS

Abstract

This proposal defines the unified streaming replication contract for every durable state in Hanzo. All sources of incremental updates — SQLite WAL, ZapDB log, blockchain state, generic append-only logs — ship frames through a single pipeline built from three orthogonal pieces:

1. One source-adapter interface (replicate.Source) — pulls frames from a backing engine.
2. One sink interface (hanzo/vfs.Backend) — the only object-store abstraction in Hanzo. All sink-side bytes are vfs.Write calls.
3. One encryption layer (luxfi/age) — frames are age-encrypted at the boundary; the master key lives in KMS per HIP-0027.

The standalone hanzoai/zapdb-replicator sidecar is deprecated and folded into hanzoai/replicate as a source adapter. The per-backend sink code in replicate/s3, replicate/gs, replicate/abs, replicate/nats, replicate/oss, replicate/sftp, replicate/webdav collapses into hanzo/vfs. One way to do everything.

This HIP does not supersede HIP-0302; HIP-0302 stays Final as the spec for the SQLite-WAL and ZapDB substrate (encryption keys, recovery objectives, K8s sidecar pattern, PQ-safe age v1.3.0+ recipients). HIP-0107 sits on top: it unifies the transport so the SQLite backend, the ZapDB backend, and any future backend route bytes through the same pipeline.

Two mechanisms, one substrate (the reconciliation). Hanzo grew TWO SQLite-HA implementations: hanzoai/replicate (a Litestream fork — continuous streaming WAL → object store, files stored as age-encrypted .zap.age) and hanzoai/vfs/replica (whole-object VACUUM INTO snapshots plus the piece Litestream lacks: a coordinator-free single-writer election). That overlap is a one-and-one-way violation. This HIP resolves it, and names the winners — one home per concern: streaming replication (hanzoai/replicate) is the canonical durability/replication transport; HRW single-writer election is hanzoai/ha (extracted from vfs/replica, which had kept storage-agnostic election inside a SQLite library — its own one-and-one-way violation); the VACUUM INTO snapshot / hydrate / adopt primitives stay in hanzoai/vfs/replica. ha decides WHO writes; vfs/replica decides HOW state ships. The whole-object snapshot path is superseded by streaming for the replication path; election (now ha) and the hydrate/adopt primitives stay. See "HA-SQLite substrate" and "Relationship to HIP-0302 and hanzoai/replicate" below.

Specification

Architecture

                ┌───────────────────────────────────────────┐
                │              hanzoai/replicate            │
                │                                           │
   ┌─────────┐  │  ┌──────────────┐    ┌─────────────────┐  │  ┌────────────┐
   │ Source  │──┼─▶│ frame-pump   │───▶│ age-encrypt     │──┼─▶│ vfs.Backend│
   │ adapter │  │  │ (one impl)   │    │ (luxfi/age)     │  │  │   (sink)   │
   └─────────┘  │  └──────────────┘    └─────────────────┘  │  └────────────┘
                └───────────────────────────────────────────┘

Four canonical source adapters ship with hanzoai/replicate:

All four implement:

package replicate

type Source interface {
    Name() string
    // PullFrames yields frames as the source produces them.
    // The returned channel closes when the source quiesces.
    PullFrames(ctx context.Context) (<-chan Frame, error)
    // Restore reads frames in order and applies them to the destination.
    Restore(ctx context.Context, frames <-chan Frame, dst string) error
}

type Frame struct {
    ID        uint64    // monotonic per-source
    Kind      FrameKind // snapshot | delta | tombstone
    Payload   []byte    // engine-specific (WAL bytes, ZAP frame, block delta, log record)
    Checksum  uint32    // CRC-32 over Payload
    Timestamp time.Time
}

Sink: hanzo/vfs is the only object store

The per-backend code currently in hanzoai/replicate/{s3,gs,abs,nats, oss,sftp,webdav} moves to hanzo/vfs as standard vfs.Backend implementations. After this HIP lands:

• hanzoai/replicate has zero direct cloud-SDK imports. The only storage interaction is vfs.Open(uri) + vfs.Backend.Write(...).
• Adding a new sink backend is one PR to hanzo/vfs. The replication pipeline picks it up for free.
• A deployment configures replication via a single vfs:// URI: vfs://gs/hanzo-replicate-mainnet/iam/<instance>/....

Encryption boundary

Age encryption happens once, at the frame boundary, before the sink call. The recipient set per HIP-0302:

• ML-KEM-768 + X25519 hybrid (age1pq prefix), age v1.3.0+
• Per-service keypair derived from a KMS-stored master key via HKDF with hanzo:replicate: domain separator
• HIP-0302 recovery objectives apply unchanged (RPO 1s SQLite / 500ms ZapDB; RTO 30s / 15s)

The sink layer sees opaque ciphertext. The sink layer must not inspect frame contents.

Frame format (canonical)

The on-disk frame format is the ZAP frame already defined in HIP-0302 §"ZapDB Streaming Replication":

Frame header (16 bytes):
  [0:4]   magic    = 0x5A415001 ("ZAP\x01")
  [4:8]   frame_id = monotonic uint32
  [8:12]  length   = payload length in bytes
  [12:14] checksum = CRC-16 of payload
  [14:15] flags    = 0x01=snapshot, 0x02=delta, 0x04=compressed
  [15:16] reserved = 0x00

The SQLite-WAL source wraps WAL frames in this envelope. The chain-state source wraps block deltas. The generic-log source wraps records. One on-disk format across all sources.

Multi-tenant bucket layout

The KMS-derived master key is per-deployment. Each tenant org gets a distinct key path:

vfs://<backend>/<bucket>/replicate/<org>/<service>/<source>/
├── snapshots/<frame_id>.snap.zap.age
├── deltas/<start>_<end>.delta.zap.age
└── latest

• <org> comes from JWT owner claim (HIP-0026 unified identity).
• <service> is the subsystem name (iam, base, kms, ...).
• <source> is one of sqlite-wal, zapdb-log, chain-state, generic-log.
• latest is a small pointer object naming the most recent frame.

Restore protocol

On pod startup, the K8s init container per HIP-0302 invokes:

hanzo-replicate restore \
  --source=sqlite-wal \
  --vfs=vfs://gs/hanzo-replicate-mainnet \
  --org=hanzo \
  --service=iam \
  --dst=/data/iam.db \
  --age-key-ref=kms://hanzo/superbase/master

The same binary handles all four source types via the --source flag. Restore drains the latest snapshot frame, then applies delta frames in monotonic order, then resumes streaming.

HA-SQLite substrate: single-writer election + adoption contract (shipped)

The streaming pipeline above answers "how do bytes get durable." It does NOT answer "which replica may write." Litestream — and therefore hanzoai/replicate — assumes a single writer and does not elect one. hanzoai/ha supplies exactly that missing half — coordinator-free single-writer election — and hanzoai/vfs/replica supplies the primitives a service needs to adopt HA without changing its own storage library. Together they are the shared substrate every Hanzo service imports instead of re-inventing durability: ha decides WHO writes, vfs/replica decides HOW state ships.

Single-writer election (Rendezvous / HRW), no coordinator — hanzoai/ha

hanzoai/ha answers, identically on every replica and with no network round-trip, "who is the single writer for key K?" via Highest-Random-Weight hashing over the live membership set (K is any ownership unit — an org id, a shard, a singleton job name):

• Owner(key, members) — deterministic winner; weight = SHA-256(key ‖ 0x00 ‖ memberID)[:8] as a big-endian uint64, tie-broken on member ID so the result is order-independent.
• IsOwner(key, selfID, members) — the per-write hot-path gate.
• Replicas(key, members, n) — owner first, then ordered failover successors, so a reader can pre-warm the key it is next in line to own.
• Membership (seam) + Static — the live-set source: a cluster impl (ha/k8s, roadmap) or single-process Static for dev and tests.
• Fail-closed: empty membership yields ok=false — never a wrong writer. No election protocol, no lock service, no service discovery, no SQL, no KV. Pure Go, stdlib only.

Ownership is PER-ORG: the owning replica writes ALL of an org's DBs (root + every per-project + per-user file) so an org's data has locality and moves as one unit on failover. Reads are served from any replica's vfs-synced copy.

Adoption contract

Every adopter wires the same three lines at its per-org store seam:

1. hydrate-on-open — Pull() (or RestoreFile) the latest durable copy before first use of an org DB, so a fresh or replaced pod starts from committed state.
2. single-writer — IsOwner(org, self, members) gates writes; a non-owner serves stale-tolerant reads (Pull-then-read) and forwards writes to the owner.
3. post-commit ship/stream — the owner ships after a write burst (the shipped Replicator.PushLoop), superseded on the streaming path by replicate continuous WAL shipping.

Shipped primitives (hanzoai/vfs/replica)

• SQLiteDB (sqlite.go) — the real per-org handle: Snapshot = VACUUM INTO a temp (forces a WAL checkpoint, one transaction-consistent copy, no torn pages, no -wal/-shm to ship); Restore = write-temp → close → drop -wal/-shm → atomic rename → reopen (a crash mid-restore leaves the old DB intact). CGO-free (modernc.org/sqlite), same durability pragmas (WAL / NORMAL / foreign_keys / busy_timeout) as base and visor.
• SnapshotFile / RestoreFile — handle-less adoption primitives so an xorm / GORM service adopts HA without a handle conflict (restore runs BEFORE the service opens its own engine).
• BackendStore (store.go) — adapts any vfs backend (file:// dev, s3:// SeaweedFS prod) to the Store, with a cheap <key>.ver content-version sidecar so a reader skips redundant pulls; Get recomputes the version authoritatively, so a stale sidecar degrades to a redundant pull, never a wrong restore.
• Per-org envelope encryption — WithEncryption(Cipher, org) seals every pushed snapshot with the org's KMS-derived key, orgID bound as AAD so a blob cannot be replayed under another org (HIP-0302 recipients).
• DBPath(org, scope, service) — the canonical HIP-0302 object layout (orgs/<org>/[scope/]<service>.db), computed identically on every replica; Version(data) — the one content hash (SHA-256) both the Replicator and any Store use.

Live adoption

visor is the reference adopter and it is LIVE. object/store_replica.go opts in via REPLICA_STORE (s3://… SeaweedFS in prod, file:// in test; unset = local-only, nil replicator, today's behavior with zero regression). It hydrates-on-open before xorm opens each org DB and runs a per-org ship loop; the success path logs, verbatim (store_replica.go:128):

visor HA: org "_global" shipping to object store (durable)

with explicit push-FAILED / push-recovered logs so a durability feature can never fail invisibly. hanzoai/cloud consolidated its own former internal/org replicator + election onto this same package (#11a3ae6 — re-exported as thin aliases in internal/org/shared.go), so cloud and visor now run ONE implementation; election has since been extracted again into hanzoai/ha (aliases repointed there; vfs/replica is now replication-only). Rolling the same adoption to iam / commerce / base is the remaining work.

Relationship to HIP-0302 and hanzoai/replicate

HIP-0302 stays Final: it owns the encryption recipients, recovery objectives, and the sidecar / emptyDir pattern. HIP-0107 owns the transport and the write-coordination on top. The two SQLite-HA mechanisms reconcile as follows — this is the decision, stated plainly:

Honest status. The election, the adoption primitives, and the visor live path are shipped. The convergence — every adopter shipping via replicate streaming rather than the Replicator whole-object push, and folding zapdb-replicator + the per-backend sinks into the one source-adapter → age → vfs.Backend pipeline (the four-source design above) — is the reconciliation roadmap this HIP ratifies, not a claim that it is already uniform. The one-and-one-way violation (two overlapping SQLite-HA libraries) is closed by DECISION here; the code convergence follows the phases below.

Implementation phases

Non-goals

• hanzoai/datastore (Hanzo Datastore fork). Datastore already uses ReplicatedMergeTree + S3-disk for its own replication. HIP-0107 does not touch it. The two systems share an S3 bucket via vfs prefix separation: s3://bucket/datastore/... is Datastore-owned; s3://bucket/replicate/... is HIP-0107-owned. No collision.
• Cross-region replication patterns. Active-active, geo-failover, read replicas across regions — separate concern. HIP-0107 ships frames to one vfs.Backend; what the operator chooses for the region/zone topology of that backend is outside scope.
• Real-time CDC for non-WAL sources. Hanzo Base's per-record hooks (HIP-0105) carry their own change-data-capture stream. That is a logical-event pipeline; HIP-0107 is a byte-level frame pipeline. They do not converge.
• Snapshot retention / GC policy. HIP-0302 already defines RPO, RTO, and retention windows. HIP-0107 inherits them unchanged.
• Replacing HIP-0302's encryption + recovery contract. HIP-0302 stays Final. HIP-0107 adds the unified transport on top.

Open questions

1. vfs URI scheme for non-S3 backends. vfs://gs/..., vfs://abs/..., vfs://sftp/... — confirm the URI prefix matches the existing hanzo/vfs convention before Phase 1.
2. Chain-state delta granularity. Per-block is the obvious default. Some Lux-derived L1s have 200ms block times — frames at 5 Hz per chain may stress the sink. Suggest configurable batching, default = 1 frame per N blocks where N is tuned per chain.
3. Generic-log fan-in. A single replicate process pumping N generic logs sharing one Source — likely fine; benchmark before committing to a goroutine-per-source model.
4. Backpressure when vfs.Backend is slow. Today zapdb-replicator drops oldest unsent frame on bounded-queue overflow. Confirm this is the policy we want across all four sources, or whether chain-state (consensus-critical) needs block-on-full instead.

References

• HIP-0027 — Secrets Management Standard (the KMS that holds the age master key per service/per org)
• HIP-0302 — Hanzo Replicate: Encrypted SQLite + ZapDB Durability for Base Services (the predecessor for the SQLite/ZapDB substrate; HIP-0107 adds the unified pipeline on top — HIP-0302 stays Final)
• HIP-0405 — S3 CRD (the hanzoai/s3 workload behind the s3:// sink this pipeline writes to in production)
• Reference implementations:
  • ~/work/hanzo/ha (SHIPPED) — github.com/hanzoai/ha, HRW single-writer election (Owner/IsOwner/Replicas/Member) + the Membership seam and Static. Pure Go, stdlib only; election with no storage dependency.
  • ~/work/hanzo/vfs/replica (SHIPPED) — the Replicator (replica.go), the real SQLiteDB snapshot/restore + SnapshotFile/RestoreFile adoption primitives (sqlite.go), and BackendStore over a vfs backend (store.go). Replication only; election lives in hanzoai/ha.
  • ~/work/hanzo/replicate — github.com/hanzoai/replicate, the Litestream fork: streaming WAL → immutable LTX → object store, files stored as age-encrypted .zap.age; the canonical replication transport.
  • ~/work/hanzo/vfs — the sink backends (vfs.Backend).
• Litestream protocol (SQLite WAL framing): https://litestream.io
• age v1.3.0+ PQ-hybrid recipients: https://age-encryption.org