bitchat/docs/BLE-ARCHITECTURE-V3.md

BLE Transport Architecture V3

The plan of record for restructuring BLEService from an 8.3k-line god object into a layered mesh stack. ARCHITECTURE_V2 rebuilt the app layer above the transport and deliberately deferred the transport itself; this document covers that remainder: what already landed, the target shape, and the order for the rest.

Why the satellite strategy stalled

V2's transport approach was to peel pure policies and closure-driven handlers out of BLEService while the class kept coordinating. The ~30 pure policy structs were a clear win. The five big handler extractions were not: each needed an "environment" of 20–30 closures that weakly capture the service and hop queues back into its state. Logic left, but state ownership and synchronization never moved, so extraction paid a plumbing tax that grew as fast as the logic shrank — the five make*HandlerEnvironment() factories alone were ~1.5k lines. The file held ~60 mutable fields across four concurrency domains whose ownership lived in comments, and every new feature added Transport requirements, state maps, and switch cases to the same class.

Two chronic costs came straight from that structure: queue-order deadlocks (the July 9 main↔bleQueue ABBA freeze), and timing-dependent tests (correctness only observable through real queues and real time).

Target shape

A packet-radio stack with one rule per layer about state and threads:

1. BLELinkLayer — the only CoreBluetooth import. Owns both managers, scanning/advertising, duty cycle, connection scheduling, MTU, write and notification backpressure buffers, state restoration. Speaks LinkEvent up (link up/down, bytes in, writable) and LinkCommand down (send bytes on link, scan/advertise policy). Knows nothing about packets, peers, or Noise. bleQueue-confined. A SimulatedLinkLayer implementing the same port gives multi-node tests real topologies with no radios and no wall-clock waits.
2. Mesh engine — one serial queue owning all protocol state: wire codec, fragmentation, dedup, relay policy, peer registry, topology, gossip sync, Noise orchestration. Synchronous single-writer logic; the pure policy satellites slot in unchanged. Endgame: the engine core becomes handle(event, now) -> [Effect] (sans-I/O), which makes the whole mesh property-testable and fuzzable in simulation.
3. Feature modules — courier, board, prekeys, private media, file transfer, voice, diagnostics, groups, verify/vouch each own their state and register for their message types. A new feature is a new module, not edits to the engine.
4. App boundary — a small Transport core both transports genuinely implement, plus capability protocols discovered with as? (MeshBridgingTransport etc.), replacing the ~90-requirement god-protocol and its inert defaults.

Concurrency contract

State is owned one of three ways:

• Engine-confined — mutated only on the serial engine queue (mesh.message). Cross-thread callers use onEngine.
• bleQueue-confined — link-layer state next to CoreBluetooth objects (link store, write/notification buffers, link-auth maps).
• Lock-backed store — state with legitimate cross-domain readers (peer registry, local identity/capabilities, traffic monitor). Writes still come from one domain; the lock exists so readers never block on a queue. Every mutation is a single whole-transition method, so readers never observe torn state.

Sync-edge order (deadlock freedom by construction, debug-enforced in onEngine):

main / test threads ──sync──▶ engine ──sync──▶ bleQueue
                                  └──sync──▶ noise / identity queues (leaves)

Nothing may sync-wait in the reverse direction: bleQueue and the crypto queues reach the engine only via async, and nothing sync-dispatches to main. Two subtleties worth knowing:

• A closure executed inside a noise-manager critical section entered from an engine slot may touch engine state directly (the blocked slot makes it exclusive) but must never sync-re-enter the engine — that is a self-deadlock.
• bleQueue critical sections (e.g. the verified-announce link rebind) must receive engine-derived values as arguments rather than fetching them through onEngine.

What landed in this pass

• Lock-backed peer state (BLEPeerRegistryStore): every main-actor Transport read (isPeerConnected, nicknames, snapshots, capability queries) reads a lock, not a queue. Runtime capability bits moved into BLELocalIdentityStateStore beside the identity they ride announces with.
• bleQueue owns the link buffers: pendingPeripheralWrites, pendingNotifications, pendingWriteBuffers are bleQueue-confined (their producers and drains already ran there); the notification drain no longer invokes CoreBluetooth from a transport queue.
• One serial engine queue: the concurrent message queue and the collections queue it guarded state with are one serial domain; every barrier flag and per-field ownership comment deleted; ~98 cross-queue hops removed. onEngine documents and debug-enforces the sync-edge order — and its trap caught two latent inversions during migration (the announce-rebind path and the noise session-generation closures).
• Capability ports: gateway/bridge/courier wiring, the panic lifecycle, and radio-state reads go through MeshBridgingTransport, PanicResettingTransport, and BluetoothStateReporting; no app code casts to BLEService anymore.
• Feature-owned state: BLEMeshPingTracker (the /ping probe map and per-link response budget) and BLEPrivateMediaSessionStore (the six generation-keyed private-media maps plus the convergence-deferral set, as whole-transition methods under a leaf lock), both with direct unit tests. The private-media store also took the last routine main-actor sync reads off the engine and turned the noise-critical-section transitions into ordinary leaf-lock calls. Remaining feature state (courier, board, prekeys) already lives in injected stores.
• Transport split: the mesh-only surface left the god-protocol. Transport is core only (lifecycle, identity, snapshots, basic messaging, noise wrappers); files/private media, voice, courier, groups, board, diagnostics, verification, and the public archive are eight capability protocols discovered with as?, alongside the bridging/panic/radio-state ports. The inert-defaults extension is gone; consumers that relied on a default keep its safe floor explicitly at the call site.
• Contract pinning: BLEQueueContractTests greps the transport sources — only onEngine may sync-enter the engine, transport code never sync-dispatches to main, and the collections queue stays deleted (waivable per line with queue-contract-ok: plus a reason).

Full suite green throughout (1,964 tests), identical wall-clock — BLE throughput is nowhere near what one serial queue sustains.

Remaining roadmap (in order)

1. Link-layer extraction. Move the CB delegates, scheduling, duty cycle, and buffers behind LinkEvent/LinkCommand ports.

   Link-auth boundary (decided): bindings become engine-owned. Today noiseAuthenticatedLinkOwners, the rebind containment rules, and the peer↔link binding maps live on bleQueue so that "check binding + auth, then act" is one critical section (the rebind path and the authenticated-send commit point in notifyOrEnqueueIfAccepted). That atomicity exists to stop a binding from changing between a security check and its action — and the engine's serial slot provides exactly the same guarantee once every rebind is an engine operation. The residual stolen-link risk is unchanged: directed payloads are Noise ciphertext, useless on a link that changed hands after the decision. Making bindings engine state also puts the receive path in its sans-I/O shape: the link layer reports received(bytes, linkID) and the engine resolves the sender binding, instead of the CB delegate resolving peers before handoff. The link layer keeps only physical link state (CB objects, connect/subscribe lifecycles, backpressure buffers) keyed by opaque link IDs.

   Extraction order: (a) the binding-free radio half — scanning, advertising, duty cycle, connection budget/scheduling — moves first (it makes no peer decisions); (b) bindings + link-auth migrate to the engine, converting readLinkState callers; (c) the delegates shrink to event emission and move behind the port.

2. Sans-I/O engine core + simulator. Make the engine formally handle(event) -> [Effect], feed it from a SimulatedLinkLayer, and move the multi-node E2E suite onto deterministic simulation (no waitUntil, no timing hygiene battles). Property tests become possible: relay-storm bounds, partition-heal convergence, dedup soundness under duplicate floods. The remaining feature code moves (courier, board, prekey, voice, file, group handlers out of the packet switch) ride this seam as handler-registered modules instead of getting closure-environment extractions now.

What this is not

No wire changes: packet formats, signing (padding is signed), the peerID identity binding, and courier tag construction are untouched — see the wire-landmines notes before assuming any of that is local.