check-perf-floors.sh is the only check between an order-of-magnitude
regression and main, and it has no tests: scripts/ carries a test file
for validate_georelays.py and two for the workflows around it, and
nothing for the script that decides whether a slow hot path merges. Its
behaviour is a shell retry loop around a Python heredoc that parses
captured test output and picks exit codes -- exactly the kind of logic
that drifts quietly.
This runs the real script against fixture PERF logs and pins the
documented contract:
- every floor met passes;
- the best value across appended attempts is what counts, so one
retry that clears the floor is enough;
- a below-floor result is re-measured and passes once it clears;
- a below-floor result on every attempt fails (exit 1);
- a floored benchmark that stops reporting fails without a retry
(exit 3) -- a re-measurement that happened to produce the line would
paper over a silently dropped benchmark;
- no PERF lines and BITCHAT_SKIP_PERF_BASELINES=1 both skip (exit 0).
The re-measure command is replaced by a stand-in that records each
invocation and appends whatever the test says to the log the gate
points it at, so the retry path runs in milliseconds with no toolchain,
and a test that retries unexpectedly fails fast instead of building the
app.
One test pins a behaviour rather than endorsing it: a benchmark that
reports a PERF line with no floor prints "consider adding a floor" and
passes. That is the gate's one advisory branch -- removing a floored
benchmark fails, adding an unfloored one does not -- and it is left as
found so the follow-up flipping it is a three-line script change with a
one-assertion test diff.
The tests run in the app matrix entry immediately before the gate they
describe. They also fall under the scripts/tests discovery in
validate_georelays.yml, which is harmless.
* Validate relay-directory changes in PR CI
The weekly automation validates a relay snapshot before it pushes its
branch, but that check lives inside the scheduled job. A change to the
reviewed relay directory that arrives as a pull request is validated by
nothing at all, which is the ground two relay-data PRs were closed on:
"PR CI doesn't run the validator, and untraceable relay-directory
updates aren't something we can merge on faith" (#1632, also #1673).
This runs the validator on a pull request touching relays/ or the
validator itself, against the base branch's reviewed copy, and publishes
the row count, unique-relay count and SHA-256 to the job summary. The
hash is the point: the relay automation already publishes one for every
snapshot it proposes, so a reviewer can now match a proposed file
against it instead of taking provenance on faith.
The gate deliberately runs the base branch's validator rather than the
pull request's copy of it. Otherwise a change that edits the data and
the validator together certifies itself -- dropping 86% of the directory
passes if the same commit relaxes MIN_BASELINE_FRACTION, and the summary
still reports "validated". The scheduled job validates from a main
checkout for the same reason. The pull request's own validator is still
exercised by the unit-test step.
A pull_request run executes the workflow definition from the pull
request's own head, so a pull request can always edit this file and
change its own gate. That path is visible in the diff, which is what
review is for; what the reviewed-validator gate closes is the quiet one.
Read-only and never pull_request_target: the job runs PR-authored code,
so it holds no write scope and no secrets. checkout is @v5 like the
other read-only PR workflows rather than SHA-pinned, since that pin
exists to protect the write-capable token in fetch_georelays.yml.
What it catches: malformed rows, out-of-range coordinates, insecure
hosts, conflicting duplicate relays, and snapshots that lose half or
double the baseline's relays. What it does not: a re-ordered file whose
rows are each valid still passes -- with a different hash, which the
summary makes visible.
The accompanying test asserts the job's security-relevant shape as text,
following the precedent in test_fetch_georelays_workflow.py: the
trigger, the absent write scope, --output never pointed at the tracked
CSV, and that the gate reads the validator from the base branch.
* Split the gate from the code under review
Codex review on d627b9f found three holes, all confirmed by rehearsal.
The gate ran the base branch's validator, but read the candidate from the
worktree and ran the pull request's unit tests first. A test file that
rewrites relays/online_relays_gps.csv at import time therefore decided
what the gate validated: the summary reported 441 rows and the hash of a
file the pull request never proposed, exit 0, while the 60-row CSV in the
commit was what would merge. The published hash is the whole argument for
reviewing relay data on evidence, so a hash that can describe different
bytes than the ones merging is worse than no hash at all.
The workflow is now two jobs with an explicit split:
- gate is the trust boundary. It checks out the base commit, so nothing
the pull request wrote is present to run, and reads the candidate as a
git object pinned to the event's head SHA instead of from a worktree.
Both inputs come from the event, so a base branch that advances while
the job is queued cannot change the verdict.
- proposed-validator runs pull-request code, so it is not a trust
boundary. It runs the unit tests and, new here, the proposed validator
against the real directory. Without that a validator change is only
ever exercised against small synthetic fixtures: capping MAX_ROWS at
400 passes the unit tests and the gate, then breaks the weekly
automation after merge with "CSV exceeds 400 data rows".
Checking out the base commit also removes last round's git show of the
validator -- at base.sha the reviewed validator is simply the one in the
workspace.
Rehearsed per case, with each branch's diff verified first: a malicious
PR that swaps the CSV from a test now exits 1 where it previously passed
green; the current automation snapshot still validates to the hash
published in #1617; a corrupt CSV, a stricter validator, and a CI-only
change all land where they should.
Replaces the scheduled workflow that pushed upstream georelay CSV directly to main unreviewed with a validator-backed automation that resolves an immutable upstream commit, validates against the reviewed baseline, and opens a non-auto-merged PR (or tracking issue) instead of writing to main. Adds a mirrored strict client-side validator in GeoRelayDirectory (all-or-nothing, size/row/host/coord caps, streamed 512KiB cap, >=50% baseline overlap) that fails safe to last-known-good, and retargets runtime refresh to bitchat's reviewed copy.
Rewrites just clean/nuke to remove only ignored build artifacts (.DerivedData/.build), never git-checkout or rm tracked project files (previously could destroy uncommitted work). Adds a CI guard (check-just-clean-safety.sh) against reintroducing source-mutating cleanup, plus README/Local.xcconfig modernization.
The macOS Debug configuration uses AppIconDebug, which had no mac-idiom
entries at all, so Debug builds shipped with the generic dock icon. And
the mac slots in AppIcon reused the full-bleed square iOS artwork; macOS
does not mask icons at render time, so even Release showed a sharp-
cornered square.
- Regenerate all AppIcon mac PNGs with the Big Sur icon grid baked in
(824x824 rounded-rect body on a transparent 1024 canvas + standard
drop shadow), derived from the same 1024 iOS art.
- Add a full mac icon set to AppIconDebug from the green debug artwork.
- Add scripts/generate-mac-appicon.swift (pure CoreGraphics) to
re-derive the mac sizes whenever the 1024 source changes.
Co-authored-by: jack <jackjackbits@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Four spurious CI failures on July 5, all loaded-runner flakiness:
- ViewSmokeTests.voiceAndMediaViews_renderAndWarmCaches asserted an exact
bin count on WaveformCache.shared for the same URL the mounted
VoiceNoteView was concurrently warming at its default 120-bin width;
whichever barrier write landed last owned the entry. Probe the cache with
a dedicated audio file no view touches, and purge both URLs. Also replace
the fixed 250ms sleep for loadDuration's background hop with a waitUntil
poll.
- sendImage_privateChatProcessesAndTransfersImage (and its sendVoiceNote /
sendImage siblings) wait on work that hops through Task.detached; the
global executor is shared with every parallel test worker, so a loaded
runner can exceed the 5s wait. Raise those positive waits to
TestConstants.longTimeout (10s) — waitUntil returns as soon as the
condition holds, so passing runs are unaffected.
- subscribeNostrEvent_addsToTimeline_ifMatchesGeohash raced concurrently
running suites (e.g. CommandProcessorTests) on the process-wide
LocationChannelManager singleton: a mid-test channel flip reroutes or
drops the event permanently, so no fixed wait recovers. The wait loop now
re-asserts the channel and redelivers the event on each poll — idempotent
because channel switches clear the processed-event set and the store
dedups by message ID — so interference heals while genuine failures still
time out.
- The performance floor gate failed on a saturated runner
(gcs.buildAndDecode at 85% of floor). check-perf-floors.sh now re-runs
the benchmark suite up to twice when a metric lands below floor,
appending to the same PERF log and keeping each benchmark's best value
across attempts: noise clears on a retry, a real algorithmic regression
fails every attempt. Floors are unchanged and never lowered by the
mechanism; missing-benchmark failures exit immediately without retrying.
Co-authored-by: jack <jackjackbits@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Package.swift's .process("Noise") resource claim silently excluded all
of bitchatTests/Noise/ from compilation since Oct 2025 - including a
complete official-vector runner (cacophony + snow XX transcripts,
transport messages, handshake hash, byte-identical to upstream).
Narrowing the resource to the JSON file and loading via Bundle.module
brings 51 Noise tests back to life, with a guard asserting each
vector's protocol name matches the app's.
CI gains a performance floor gate: perf-floors.json carries deliberately
generous floors (~25% of measured throughput) that catch algorithmic
regressions without flaking on runner variance; PERF lines reach the
gate via an O_APPEND side-channel file since swift test --parallel
swallows passing tests' stdout.
Tests are now hermetic: FavoritesPersistenceService uses an in-memory
keychain under test (fixes the securityd hang that blocked pipeline
benchmarks locally) and read-receipt persistence uses a wiped scratch
UserDefaults suite instead of .standard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>