Link layer slice 4: deterministic multi-node mesh simulation (and the panic-announce bug it caught) (#1548)

* Cohere per-link Noise auth and rebind containment into BLELinkAuthState

The authenticated-link owners, the reconnect revalidation policy, and
the two rebind-containment cooldowns were four loose bleQueue-owned
maps whose invariants lived in call-site discipline: every teardown
path had to remember to retire the proof AND close the revalidation
epoch (the pair appeared seven times), and both cooldowns hand-rolled
the same prune-check-record dance. BLELinkAuthState owns them as whole
transitions — retireLink, retireLinks(ownedBy:), permitRebind,
permitRedundantRetirement — with the ownership question (bleQueue
today, engine after the option-B flip) answered in one place.

No behavior change; the one call-site reordering (redundant retirement
computes the survivor before the cooldown check instead of after) is
outcome-equivalent since the cooldown only ever recorded when a
survivor existed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Split identity-link bindings out of the physical link store

BLELinkStateStore owned two different kinds of truth: what physical
links exist (CB handles, connect lifecycles, characteristics, stream
assemblers) and who each link belongs to (peer bindings in both roles
plus the preferred-peripheral reverse map for directed sends and fanout
collapse). The bindings now live on BLELinkBindings — same bleQueue
ownership, whole-transition methods, direct tests for the rotation
reverse-map cleanup and the preferred-link survivor repair that were
previously only exercised end to end. Composed operations that need
both truths (remove-with-repair, direct link state, the subscribed-
central snapshot, bind-only-live-links) live on the transport as
explicitly bleQueue-confined helpers.

This is the structural half of the option-B boundary flip
(docs/BLE-ARCHITECTURE-V3.md): ownership of the bindings can now move
to the engine without touching what-links-exist. An audit of every
physical clear/remove found three sites (emergency clear, both
unauthorized branches) that needed explicit binding-clear pairing under
the split — each now clears both.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix iOS-gated constructors and preserve containment cooldowns on reset

CI caught what the macOS SwiftPM build cannot see: two #if os(iOS)
sites still passed the peerID field that slice B1 removed from
BLEPeripheralLinkState (willRestoreState in BLEService and
armPendingBackgroundConnects in BLERadioController). Both fixed and
verified with a local iOS simulator xcodebuild.

Codex also caught a real regression: BLELinkAuthState.removeAll()
cleared the rebind/retirement cooldown maps, which the original panic
and emergency reset paths deliberately left alive. A stable
CoreBluetooth UUID must not earn a fresh rebind allowance just because
the session state around it was wiped. removeAll() now clears only the
proofs and revalidation epochs, and BLELinkAuthStateTests pins the
survival invariant along with the other auth-state transitions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Link layer slice 3: the option-B domain flip — bindings and link-auth move to the engine

The identity domain (BLELinkBindings + BLELinkAuthState) is now owned
by the engine queue, with a DEBUG dispatchPrecondition trapping any
access from another queue. bleQueue keeps only physical link state.

What changed shape:

- Receive path is sans-I/O: bleQueue decodes frames and hands
  (packet, linkID) up through ingestDecodedPacket (panic lifecycle
  captured at the handoff); attributeAndHandlePacket resolves the
  sender binding, rejects spoofed senders, applies raw-announce
  binding, and records ingress on the engine. Per-link frame order is
  preserved end to end (both queues serial), which supersedes the old
  batch-local TOCTOU binding in the notification path.
- The rotation rebind is one engine slot: containment checks, proof
  retirement, binding flip, reconnect decision, and rotated-identity
  retirement run straight-line; only CoreBluetooth cancels hop to
  bleQueue. The engine->bleQueue->engine ping-pong is gone, along with
  the _test_afterVerifiedDirectRebindEnqueued pause hook — the test
  that used it now asserts the atomicity directly (a paused engine
  wedged the old gate design into a three-queue deadlock).
- Authenticated-send eligibility (notifyOrEnqueueIfAccepted,
  writeOrEnqueueIfAccepted) is checked on the engine, serialized
  against rebinds by construction; only physical admission
  (updateValue/write/backpressure) runs on bleQueue.
- Teardown splits into discardPeripheralLinkPhysical (bleQueue, inline
  in the delegates) + retirePeripheralLinkIdentity (engine hop with
  survivor repair reading liveness via readLinkState). A binding can
  briefly outlive its physical link; liveness queries join against the
  physical store and the queued retirement converges the two.
- Gossip delegate sends enter the engine via onEngine — safe because
  mesh.sync sits above the engine in the sync order (production engine
  code only async-dispatches into the manager).
- checkPeerConnectivity rides an engine slot from the bleQueue
  maintenance tick.

No wire changes. 1,974 tests green (parallel and serial), iOS
simulator build clean, Periphery clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Link layer slice 4: deterministic multi-node mesh simulation — and the panic-announce bug it caught

SimulatedMesh wires real CoreBluetooth-free BLEService engines
edge-to-edge through the outbound packet tap and _test_ingestFrame
(the production attribution path the B2 flip created), with per-edge
synthetic link IDs and manual-scheduler time. Five multi-node tests
run in ~40ms with no wall-clock waits:

- announce exchange binds simulated links and connects peers
- Noise sessions establish end-to-end (real crypto, both directions)
- a public message relays across a line topology inside a TTL/frame
  budget (storm bound asserted)
- an 8x duplicate flood delivers exactly once
- a panic rotation rebinds the survivor's link exactly once and stays
  — the scenario that previously needed two phones and log archaeology

Fidelity boundary (documented in the harness): no physical links, so
fanout planning and backpressure are not exercised; attribution,
binding, dedup, TTL, relay decisions, and sessions are the real
engine code.

The simulator found a real bug on its first run: the forced-announce
throttle's lastSent survived a panic, so a rotation within
bleForceAnnounceMinIntervalSeconds of the last announce silently
swallowed the new identity's announce — leaving it invisible to the
mesh until the next maintenance cycle. Today's device test only
passed because the previous announce happened to be minutes old.
BLEAnnounceThrottle gains reset(), called from the panic slot so the
rotated identity owes no throttle debt; pinned by a unit test and the
mesh rotation test.

New DEBUG seams: _test_ingestFrame (production ingress attribution),
_test_forceAnnounce, _test_fenceEngine.

1,980 tests green, Periphery clean, iOS simulator build clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: jack <jackjackbits@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jack 2026-07-30 01:14:19 +01:00 committed by GitHub
parent 2f5b56ce57
commit cdebdd9347
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
7 changed files with 377 additions and 8 deletions

View File

@ -94,7 +94,6 @@
isa = PBXFileSystemSynchronizedBuildFileExceptionSet;
membershipExceptions = (
Info.plist,
bitchatShareExtension.entitlements,
);
target = 57CA17A36A2532A6CFF367BB /* bitchatShareExtension */;
};
@ -379,6 +378,11 @@
E0A1B2C3D4E5F6012345678D /* relays/online_relays_gps.csv in Resources */,
);
};
7E9B64F63F93443FB7BA12DF /* Resources */ = {
isa = PBXResourcesBuildPhase;
files = (
);
};
C5E027A42ECCDFD700BD6012 /* Resources */ = {
isa = PBXResourcesBuildPhase;
files = (
@ -395,13 +399,6 @@
E0A1B2C3D4E5F6012345678E /* relays/online_relays_gps.csv in Resources */,
);
};
7E9B64F63F93443FB7BA12DF /* Resources */ = {
isa = PBXResourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXResourcesBuildPhase section */
/* Begin PBXSourcesBuildPhase section */

View File

@ -37,4 +37,13 @@ final class BLEAnnounceThrottle: @unchecked Sendable {
return true
}
}
/// Forgets the last-sent timestamp. A panic rotation calls this so the
/// new identity's first announce cannot be swallowed by the old
/// identity's throttle debt otherwise a panic within the forced
/// minimum interval of the last announce leaves the rotated identity
/// invisible until the next maintenance cycle.
func reset() {
lock.withLock { lastSent = .distantPast }
}
}

View File

@ -782,6 +782,11 @@ final class BLEService: NSObject {
// rebind/retirement cooldowns deliberately survive (see
// BLELinkAuthState.removeAll).
linkAuth.removeAll()
// The new identity owes no announce-throttle debt: without this,
// a panic within the forced minimum interval of the last
// announce swallows the rotation announce and the new identity
// stays invisible until the next maintenance cycle.
announceThrottle.reset()
// These callbacks belong to pre-panic transfer state. Invoking
// them would let queued UI work recreate or resend wiped media.
privateMediaSessions.panicReset()
@ -3353,6 +3358,27 @@ extension BLEService {
}
}
/// Simulated-link ingress: the full production attribution path
/// binding lookup, spoof rejection, raw-announce binding, ingress
/// recording for a frame arriving on a synthetic link. The
/// SimulatedMesh harness feeds every node through this, so multi-node
/// tests exercise the same engine code as CoreBluetooth ingress.
func _test_ingestFrame(_ packet: BitchatPacket, link: BLEIngressLinkID) {
ingestDecodedPacket(packet, link: link, linkDescription: "Simulated \(link)")
}
/// Sends an unthrottled announce, exactly like the maintenance forced
/// path. SimulatedMesh uses this as the deterministic discovery step.
func _test_forceAnnounce() {
onEngine { sendAnnounceNow(forceSend: true) }
}
/// Blocks until every engine slot enqueued so far has run the
/// deterministic settling fence for simulated-mesh pumping.
func _test_fenceEngine() {
onEngine {}
}
func _test_emitTransportEvent(
_ event: TransportEvent,
completion: @escaping () -> Void,

View File

@ -68,6 +68,21 @@ struct BLEAnnounceThrottleTests {
#expect(accepted.value == 1)
#expect(throttle.elapsed(since: now.addingTimeInterval(3)) == 3)
}
@Test
func resetForgetsThrottleDebtSoARotationAnnounceIsNeverSwallowed() {
let throttle = BLEAnnounceThrottle(
normalMinimumInterval: 1,
forcedMinimumInterval: 1
)
let now = Date()
#expect(throttle.shouldSend(force: true, now: now))
// A panic inside the forced window would be throttled...
#expect(!throttle.shouldSend(force: true, now: now.addingTimeInterval(0.2)))
// ...so the rotation resets the debt and announces immediately.
throttle.reset()
#expect(throttle.shouldSend(force: true, now: now.addingTimeInterval(0.3)))
}
}
private final class LockedCounter: @unchecked Sendable {

View File

@ -0,0 +1,129 @@
import BitFoundation
import Foundation
@testable import bitchat
/// A deterministic multi-node mesh over real `BLEService` engines and no
/// CoreBluetooth: nodes are wired edge-to-edge through the outbound packet
/// tap and the production ingress-attribution path (`_test_ingestFrame`),
/// so announces bind links, signatures verify, Noise handshakes complete,
/// and rotation rebinds run exactly the engine code a radio would drive.
///
/// Determinism model: outbound packets are buffered under a lock (the tap
/// fires on each sender's engine); the test thread pumps deliveries and
/// fences every engine between rounds. Timer-driven work (relay jitter,
/// deferred flushes) is released explicitly through each node's
/// `BLEEngineManualScheduler` via `advanceTime`.
///
/// Fidelity boundary: there are no physical links, so per-link fanout
/// planning always reports failure to the sender (directed packets spool)
/// every capture happens at the pre-planning tap. Protocol-level
/// behavior (attribution, binding, dedup, TTL, relay decisions, sessions)
/// is faithful; link-selection and backpressure behavior is not exercised.
final class SimulatedMesh {
struct Node {
let service: BLEService
let scheduler: BLEEngineManualScheduler
}
private let lock = NSLock()
private var pendingDeliveries: [(from: Int, packet: BitchatPacket)] = []
/// Total (packet, receiving-node) deliveries pumped the storm bound.
private(set) var deliveredFrameCount = 0
private(set) var nodes: [Node] = []
private var neighbors: [Set<Int>] = []
@discardableResult
func addNode(nickname: String) -> Node {
let keychain = MockKeychain()
let identityManager = MockIdentityManager(keychain)
let idBridge = NostrIdentityBridge(keychain: MockKeychainHelper())
let scheduler = BLEEngineManualScheduler()
let service = BLEService(
keychain: keychain,
idBridge: idBridge,
identityManager: identityManager,
initializeBluetoothManagers: false,
engineScheduler: scheduler
)
let index = nodes.count
let node = Node(service: service, scheduler: scheduler)
nodes.append(node)
neighbors.append([])
service.setNickname(nickname)
service._test_onOutboundPacket = { [weak self] packet in
// Runs on the sender's engine; only buffer here delivering
// inline would nest one engine inside another.
guard let self else { return }
self.lock.lock()
self.pendingDeliveries.append((from: index, packet: packet))
self.lock.unlock()
}
return node
}
func connect(_ a: Int, _ b: Int) {
neighbors[a].insert(b)
neighbors[b].insert(a)
}
/// The synthetic link a frame from `sender` arrives on at `receiver`.
/// Stable per directed edge, like a CoreBluetooth central UUID.
func linkUUID(from sender: Int, at receiver: Int) -> String {
"SIM-\(sender)-TO-\(receiver)"
}
func forceAnnounce(from index: Int) {
nodes[index].service._test_forceAnnounce()
pump()
}
/// Pumps buffered deliveries until the mesh is quiescent: no pending
/// frames and every engine drained. Timer-deferred work stays pending
/// until `advanceTime`.
func pump(maxRounds: Int = 64) {
for _ in 0..<maxRounds {
lock.lock()
let batch = pendingDeliveries
pendingDeliveries.removeAll()
lock.unlock()
if batch.isEmpty {
// Engines may still be running slots that will emit more.
nodes.forEach { $0.service._test_fenceEngine() }
lock.lock()
let stillEmpty = pendingDeliveries.isEmpty
lock.unlock()
if stillEmpty { return }
continue
}
for (from, packet) in batch {
for receiver in neighbors[from] {
deliveredFrameCount += 1
nodes[receiver].service._test_ingestFrame(
packet,
link: .central(linkUUID(from: from, at: receiver))
)
}
}
nodes.forEach { $0.service._test_fenceEngine() }
}
fatalError("SimulatedMesh.pump did not quiesce in \(maxRounds) rounds — relay storm?")
}
/// Advances every node's engine clock (releasing relay jitter, retries,
/// deferred flushes) and pumps the resulting traffic.
func advanceTime(by interval: TimeInterval) {
nodes.forEach { $0.scheduler.advance(by: interval) }
pump()
}
/// Full discovery round: every node announces, traffic settles.
func announceAll() {
for node in nodes {
node.service._test_forceAnnounce()
}
pump()
}
}

View File

@ -0,0 +1,172 @@
import BitFoundation
import Foundation
import Testing
@testable import bitchat
/// Deterministic multi-node mesh tests over the real engine (no
/// CoreBluetooth, no wall-clock waits): announces bind simulated links,
/// signatures verify, Noise sessions establish, and rotation rebinds run
/// the same engine slots a radio would drive. See SimulatedMesh for the
/// fidelity boundary.
@Suite(.serialized)
struct SimulatedMeshTests {
@Test
func announceExchangeBindsLinksAndConnectsPeers() {
let mesh = SimulatedMesh()
let a = mesh.addNode(nickname: "alice")
let b = mesh.addNode(nickname: "bob")
mesh.connect(0, 1)
mesh.announceAll()
// Raw direct announces bind each directed edge to the sender.
#expect(b.service._test_centralBinding(mesh.linkUUID(from: 0, at: 1)) == a.service.myPeerID)
#expect(a.service._test_centralBinding(mesh.linkUUID(from: 1, at: 0)) == b.service.myPeerID)
// Verified announces register connected peers on both sides.
#expect(a.service.getConnectedPeers().contains(b.service.myPeerID))
#expect(b.service.getConnectedPeers().contains(a.service.myPeerID))
}
@Test
func noiseSessionEstablishesEndToEnd() {
let mesh = SimulatedMesh()
let a = mesh.addNode(nickname: "alice")
let b = mesh.addNode(nickname: "bob")
mesh.connect(0, 1)
mesh.announceAll()
// Handshake initiation and any deferred retries ride engine timers.
mesh.advanceTime(by: 2)
#expect(a.service.canDeliverSecurely(to: b.service.myPeerID))
#expect(b.service.canDeliverSecurely(to: a.service.myPeerID))
}
@Test
func publicMessageRelaysAcrossLineTopologyWithinTTLBudget() async {
let mesh = SimulatedMesh()
let a = mesh.addNode(nickname: "alice")
_ = mesh.addNode(nickname: "bob")
let c = mesh.addNode(nickname: "carol")
mesh.connect(0, 1)
mesh.connect(1, 2)
mesh.announceAll()
mesh.advanceTime(by: 2)
let baseline = mesh.deliveredFrameCount
let capture = TransportEventCapture()
c.service.eventDelegate = capture
a.service.sendMessage("hello line", mentions: [])
mesh.pump()
// Relay jitter defers B's forward; release it (twice: the relay's
// own broadcast may schedule follow-on work).
mesh.advanceTime(by: 2)
mesh.advanceTime(by: 2)
let arrived = await capture.drainedPublicMessageCount(content: "hello line") == 1
#expect(arrived)
// Storm bound: a single public message across one relay hop must
// not multiply into more than a handful of frames.
#expect(mesh.deliveredFrameCount - baseline <= 12)
}
@Test
func duplicateFloodIsDeliveredOnce() async {
let mesh = SimulatedMesh()
let a = mesh.addNode(nickname: "alice")
let b = mesh.addNode(nickname: "bob")
mesh.connect(0, 1)
mesh.announceAll()
let capture = TransportEventCapture()
b.service.eventDelegate = capture
let packet = BitchatPacket(
type: MessageType.message.rawValue,
senderID: Data(hexString: a.service.myPeerID.id) ?? Data(),
recipientID: nil,
timestamp: UInt64(Date().timeIntervalSince1970 * 1000),
payload: Data("flooded".utf8),
signature: nil,
ttl: TransportConfig.messageTTLDefault
)
let signed = a.service.signPacketForBroadcast(packet)
let link = BLEIngressLinkID.central(mesh.linkUUID(from: 0, at: 1))
for _ in 0..<8 {
b.service._test_ingestFrame(signed, link: link)
}
mesh.pump()
mesh.advanceTime(by: 2)
let deliveredOnce = await capture.drainedPublicMessageCount(content: "flooded") == 1
#expect(deliveredOnce)
}
@Test
func panicRotationRebindsSurvivorExactlyOnceAndStays() {
let mesh = SimulatedMesh()
let a = mesh.addNode(nickname: "alice")
let b = mesh.addNode(nickname: "bob")
mesh.connect(0, 1)
mesh.announceAll()
mesh.advanceTime(by: 2)
let oldBobID = b.service.myPeerID
let bobLinkOnAlice = mesh.linkUUID(from: 1, at: 0)
#expect(a.service._test_centralBinding(bobLinkOnAlice) == oldBobID)
// Bob panics: the production sequence suspend, rotate the whole
// identity, commit over the same simulated link.
b.service.suspendForPanicReset()
b.service.resetIdentityForPanic(currentNickname: "anon", restartServices: false)
b.service.completePanicReset(restartServices: false)
mesh.pump()
let newBobID = b.service.myPeerID
#expect(newBobID != oldBobID)
// His first verified direct announce heals the stale binding in
// one engine slot on the survivor.
mesh.forceAnnounce(from: 1)
mesh.advanceTime(by: 2)
#expect(a.service._test_centralBinding(bobLinkOnAlice) == newBobID)
// Containment: further announces (and the rebind cooldown) leave
// the healed binding alone no flip-flop back to the dead ID.
mesh.forceAnnounce(from: 1)
mesh.forceAnnounce(from: 0)
mesh.advanceTime(by: 2)
#expect(a.service._test_centralBinding(bobLinkOnAlice) == newBobID)
#expect(a.service.getConnectedPeers().contains(newBobID))
}
}
/// Captures `.publicMessageReceived` transport events. Delivery crosses the main
/// actor (`notifyUI`), so counting first drains that hop a bounded number
/// of main-actor round-trips, never a wall-clock wait (the mesh is already
/// quiescent when this is called; only the queued MainActor task remains).
private final class TransportEventCapture: TransportEventDelegate, @unchecked Sendable {
private let lock = NSLock()
private var publicMessages: [String] = []
func didReceiveTransportEvent(_ event: TransportEvent) {
guard case let .publicMessageReceived(_, _, content, _, _) = event else { return }
lock.lock()
publicMessages.append(content)
lock.unlock()
}
private func count(content: String) -> Int {
lock.lock()
defer { lock.unlock() }
return publicMessages.filter { $0 == content }.count
}
func drainedPublicMessageCount(content: String, drains: Int = 50) async -> Int {
for _ in 0..<drains {
if count(content: content) > 0 { break }
await MainActor.run {}
}
return count(content: content)
}
}

View File

@ -201,6 +201,27 @@ throughput is nowhere near what one serial queue sustains.
packet switch) ride this seam as handler-registered modules instead
of getting closure-environment extractions now.
**The simulator half is done — simulator-first.** Because the B2
receive path already hands `(packet, linkID)` up through one choke
point, `SimulatedMesh` (bitchatTests/Simulation/) wires real
CB-free `BLEService` engines edge-to-edge through the outbound tap
and `_test_ingestFrame` (the production attribution path), with
per-edge synthetic link IDs and manual-scheduler time. Five
deterministic multi-node tests run in ~40ms: announce/bind
convergence, end-to-end Noise establishment, line-topology relay
within a TTL/frame budget, duplicate-flood dedup, and the panic
rotation single-slot rebind + containment — the scenario that
previously required two phones. Fidelity boundary: no physical
links, so fanout planning/backpressure is not exercised; protocol
behavior is. On its first day the simulator found a real bug: the
forced-announce throttle survived panic, so a rotation within
`bleForceAnnounceMinIntervalSeconds` of the last announce left the
new identity invisible until the next maintenance cycle
(`BLEAnnounceThrottle.reset()` now runs in the panic slot).
Remaining from the original slice-C scope: the mechanical delegate
extraction behind explicit LinkEvent/LinkCommand types, and the
formal `handle(event) -> [Effect]` engine shape.
## What this is not
No wire changes: packet formats, signing (padding is signed), the