R2 Design — Distributed Ownership (fencing + handle registry)

Date: 2026-08-18 · Status: implemented in tree, not deployed · ROADMAP-2026-08-10 §R2 One line: a write lease is a JSON grant (client_id + session_id + fencing_epoch), not hostname-PID; every file-data commit CASes that grant so a partitioned old writer cannot publish after a new writer holds the inode.

0. Why this exists

The G6 exclusive-writer lease is a TTL lock, not fencing.

1. Holder is hostname-PID (swiftfuse-cli mount). Daemon restart → new PID → 30s EBUSY even though it is the same machine.
2. acquire_write_lease PUTs that string under {prefix}/leases/{inode} attached to an etcd lease. Re-entrancy is byte-equality on the string.
3. commit_inode CASes inode.version + etcd mod_revision only. A partitioned old writer that still holds a stale Handle can commit after a new writer has the lease.
4. There is no distributed open-handle table. Cross-node open-unlinked GC is local NodeTable only.

R2 does not add INVAL_ENTRY / INVAL_INODE / NOTIFY_STORE, does not flip write_back, and does not use FOPEN_DIRECT_IO. JSON-bytes CAS stays on mod_revision.

1. Identities

session_id changing with the same client_id is the restart case: re-acquire is allowed and must bump fencing_epoch so in-flight commits from the dead session are fenced.

2. Lease record (etcd key {prefix}/leases/{inode})

JSON, schema_version = 1:

{
  "schema_version": 1,
  "client_id": "swift1-a1b2c3d4-...",
  "session_id": "sess-7f8e...",
  "fencing_epoch": 3,
  "etcd_lease_id": 75849201023
}

Backward compatible: a pre-R2 value that is a bare UTF-8 string is parsed as

client_id = that string, session_id = "", fencing_epoch = 0, etcd_lease_id = 0

The next acquire from that holder upgrades the key to JSON.

Acquire (acquire_write_lease_v2)

1. Grant a new etcd lease. Try Compare::version(lease_key) == 0 PUT of a JSON record with fencing_epoch = 1.
2. If the key exists:
   • other client_id → None (FUSE maps to EBUSY). Surplus grant revoked.
   • same client_id + same session_id → return the existing grant (no epoch bump). Surplus grant revoked.
   • same client_id + new session_id → PUT replacement JSON with fencing_epoch + 1 attached to the new etcd lease (CAS on the bytes we just read). This is restart.
3. acquire_write_lease(inode, holder) stays. It calls v2 with client_id = session_id = holder and returns bool. Existing client-A / client-B tests stay valid.

Sqlite: acquire always succeeds (single-node). An in-memory map still tracks the latest grant so commit_inode_fenced can reject a stale epoch. Other-client EBUSY is etcd-only.

Grant stored on the FUSE Handle

LeaseGrant { client_id, session_id, fencing_epoch }

Set at open / create after a successful acquire. flush_handle and the O_TRUNC commit pass this grant into commit_inode_fenced.

3. Fencing on file-data commit

commit_inode is unchanged: version + mod_revision only. Directory mtime touches, chmod, and the grow-only SETATTR helper (no write lease) stay on this path.

commit_inode_fenced(meta, grant) is the write path. It must succeed only if all of:

• stored inode.version == meta.version
• lease key exists
• lease.client_id == grant.client_id
• lease.fencing_epoch == grant.fencing_epoch

Otherwise CommitResult::Conflict (never Committed).

On etcd this is one txn:

WHEN  inode.mod_revision == expected
 AND  lease.value        == bytes we just read (after the epoch/client check)
THEN  PUT inode (version+1)

Lease missing, other holder, or stale epoch fails the compare → Conflict. A partitioned old writer with epoch N cannot commit after epoch N+1 is on the key.

Wired at:

Pure check (no etcd) lives in swift_meta::lease::lease_allows_commit.

4. Handle registry (minimum viable)

Keys: {prefix}/handles/{inode}/{client_id}/{fh}

• Value: "1" (presence only).
• Attached to a daemon session etcd lease (TTL 60s, keepalive ~20s). Daemon death expires every handle this mount registered.
• register_handle on every open / create.
• unregister_handle on release.
• handle_count(inode) = prefix GET count.

Sqlite: in-memory HashMap<(inode, client_id, fh), ()>.

GC hook — wired

remove_file_entry must consult the registry before unlink_inode. A mount that never opened the file used to delete the record under a peer fd. If handle_count>0 or lease_held, only the name is dropped.

ChunkFs::maybe_gc does not AND handle_count onto the old local gc_ready. That would leak: the unlinking mount never sees another FORGET, and the holding mount never marked deleted.

Full condition:

this mount:     open==0 ∧ lookup_count==0
engine:         handle_count==0 ∧ !lease_held(inode)
name:           sqlite tombstone `deleted:/{id}`, or lookup(parent, name)
                missing / points at another inode (etcd)

Engine errors skip GC. After the last peer close+forget, that peer’s maybe_gc reclaims (nlink is not stored; name-gone is the distributed deleted bit). Tests: existing open-unlinked suite + remote_handle_blocks_deferred_gc + writer_lease_blocks_deferred_gc.

5. What this change does not do

• Live partition gate (tc netem + old writer resume). That is the ROADMAP R2 gate, not this unit of work.
• Live kernel FORGET timing. Remount runs sweep_unlinked for nameless leftovers.
• Deploy, remount, kernel_ttl flip, pkill, cargo clean.
• Notify / direct-IO / write-through workarounds.

6. Tests (must exist)

7. Files

• crates/swift-meta/src/lease.rs — record, grant, parse, lease_allows_commit
• crates/swift-meta/src/{engine,etcd,sqlite,lib,model}.rs
• crates/swift-fuse/src/{chunk_fs,inode_fs}.rs
• crates/swiftfuse-cli/src/main.rs — stable client_id
• crates/swift-meta/tests/{etcd_test,engine_test}.rs