7 Commits

Author SHA1 Message Date
Jozef Koval
739077305f Exit 2 on a malformed floors file instead of re-measuring
A broken floors file raised an uncaught exception, so Python exited 1.
The loop read exit 1 as "below floor" and measured twice more before
it reported a real regression. The gate now names the file and exits
2. The loop does not retry exit 2, as its comment says for setup
errors.

Broken means unreadable, not valid JSON, no "floors" object, or a
floor that is not a finite number. Some of these did not crash before,
but gave a wrong result. A NaN floor let every value pass, because a
comparison with NaN is always false. A `true` floor counted as 1, and
a `null` floor was reported as NO-FLOOR with exit 3.
test_malformed_floors_file_fails_without_a_retry covers six cases.

The exit-code comments also called exit 3 "floored benchmark missing".
This branch fails unfloored benchmarks with exit 3 too, so they now
say "floors and benchmarks out of step".
2026-11-01 17:56:30 +01:00
Jozef Koval
a2061b07d3 Log the clearing value first in the best-value gate test
The test logged 50, then 500. The last value was also the best one, so
a gate that kept the last value instead of the best passed all eight
tests. With 500 first and 50 last, that gate stays below the floor and
fails this test.
2026-11-01 17:29:00 +01:00
Jozef Koval
54e691e419 Import future annotations so the gate tests load on Python 3.9
The test file annotates with `str | None` and `Path | None`. Python
evaluates these when it defines each function, and Python 3.9, the
stock macOS python3, has no `|` on types. So the import raised a
TypeError, and discovery of the whole scripts/tests folder failed.

With the future import, Python keeps the annotations as strings and
does not evaluate them.
2026-11-01 17:28:41 +01:00
Jozef Koval
217b522ec3 Fail the floor gate on a benchmark with no floor
The gate fails when a floored benchmark stops reporting (exit 3, no
retry) so that a benchmark cannot be dropped without an explicit
floors-file change. The other direction was advisory: a benchmark that
reports a PERF line with no floor printed "consider adding a floor"
into the job log and passed. Nothing else surfaces that line, so a new
hot-path benchmark could measure on every run and guard nothing, and
the file the gate reads would still look complete.

That is the shape of the localization gap closed in #1654, where a key
missing from every catalog compiled and shipped English while CI stayed
green, and the move is the one made in #1646 and #1648: an advisory
check becomes blocking once its backlog is zero. perf-floors.json
carries 12 floors and PerformanceBaselineTests has 12 benchmarks, so
flipping this costs nothing today and holds the two in step from now
on.

An unfloored benchmark now fails alongside a missing one, with the same
exit code and for the same reason: the floors file and the benchmark
suite must change together, and no re-measurement can repair a drift
between them. The message names the file to edit.

The test pinned in the previous commit flips from "reported but does
not fail" to "fails without a retry"; the philosophy note in the floors
file says "added, renamed or removed" instead of "renamed or removed".
2026-09-24 22:34:02 +02:00
Jozef Koval
cd8b274f89 Pin the performance floor gate's contract with tests
check-perf-floors.sh is the only check between an order-of-magnitude
regression and main, and it has no tests: scripts/ carries a test file
for validate_georelays.py and two for the workflows around it, and
nothing for the script that decides whether a slow hot path merges. Its
behaviour is a shell retry loop around a Python heredoc that parses
captured test output and picks exit codes -- exactly the kind of logic
that drifts quietly.

This runs the real script against fixture PERF logs and pins the
documented contract:

- every floor met passes;
- the best value across appended attempts is what counts, so one
  retry that clears the floor is enough;
- a below-floor result is re-measured and passes once it clears;
- a below-floor result on every attempt fails (exit 1);
- a floored benchmark that stops reporting fails without a retry
  (exit 3) -- a re-measurement that happened to produce the line would
  paper over a silently dropped benchmark;
- no PERF lines and BITCHAT_SKIP_PERF_BASELINES=1 both skip (exit 0).

The re-measure command is replaced by a stand-in that records each
invocation and appends whatever the test says to the log the gate
points it at, so the retry path runs in milliseconds with no toolchain,
and a test that retries unexpectedly fails fast instead of building the
app.

One test pins a behaviour rather than endorsing it: a benchmark that
reports a PERF line with no floor prints "consider adding a floor" and
passes. That is the gate's one advisory branch -- removing a floored
benchmark fails, adding an unfloored one does not -- and it is left as
found so the follow-up flipping it is a three-line script change with a
one-assertion test diff.

The tests run in the app matrix entry immediately before the gate they
describe. They also fall under the scripts/tests discovery in
validate_georelays.yml, which is harmless.
2026-09-24 22:31:49 +02:00
Jozef Koval
eae01ef9d8
Validate relay-directory changes in PR CI (#1711)
* Validate relay-directory changes in PR CI

The weekly automation validates a relay snapshot before it pushes its
branch, but that check lives inside the scheduled job. A change to the
reviewed relay directory that arrives as a pull request is validated by
nothing at all, which is the ground two relay-data PRs were closed on:
"PR CI doesn't run the validator, and untraceable relay-directory
updates aren't something we can merge on faith" (#1632, also #1673).

This runs the validator on a pull request touching relays/ or the
validator itself, against the base branch's reviewed copy, and publishes
the row count, unique-relay count and SHA-256 to the job summary. The
hash is the point: the relay automation already publishes one for every
snapshot it proposes, so a reviewer can now match a proposed file
against it instead of taking provenance on faith.

The gate deliberately runs the base branch's validator rather than the
pull request's copy of it. Otherwise a change that edits the data and
the validator together certifies itself -- dropping 86% of the directory
passes if the same commit relaxes MIN_BASELINE_FRACTION, and the summary
still reports "validated". The scheduled job validates from a main
checkout for the same reason. The pull request's own validator is still
exercised by the unit-test step.

A pull_request run executes the workflow definition from the pull
request's own head, so a pull request can always edit this file and
change its own gate. That path is visible in the diff, which is what
review is for; what the reviewed-validator gate closes is the quiet one.

Read-only and never pull_request_target: the job runs PR-authored code,
so it holds no write scope and no secrets. checkout is @v5 like the
other read-only PR workflows rather than SHA-pinned, since that pin
exists to protect the write-capable token in fetch_georelays.yml.

What it catches: malformed rows, out-of-range coordinates, insecure
hosts, conflicting duplicate relays, and snapshots that lose half or
double the baseline's relays. What it does not: a re-ordered file whose
rows are each valid still passes -- with a different hash, which the
summary makes visible.

The accompanying test asserts the job's security-relevant shape as text,
following the precedent in test_fetch_georelays_workflow.py: the
trigger, the absent write scope, --output never pointed at the tracked
CSV, and that the gate reads the validator from the base branch.

* Split the gate from the code under review

Codex review on d627b9f found three holes, all confirmed by rehearsal.

The gate ran the base branch's validator, but read the candidate from the
worktree and ran the pull request's unit tests first. A test file that
rewrites relays/online_relays_gps.csv at import time therefore decided
what the gate validated: the summary reported 441 rows and the hash of a
file the pull request never proposed, exit 0, while the 60-row CSV in the
commit was what would merge. The published hash is the whole argument for
reviewing relay data on evidence, so a hash that can describe different
bytes than the ones merging is worse than no hash at all.

The workflow is now two jobs with an explicit split:

- gate is the trust boundary. It checks out the base commit, so nothing
  the pull request wrote is present to run, and reads the candidate as a
  git object pinned to the event's head SHA instead of from a worktree.
  Both inputs come from the event, so a base branch that advances while
  the job is queued cannot change the verdict.
- proposed-validator runs pull-request code, so it is not a trust
  boundary. It runs the unit tests and, new here, the proposed validator
  against the real directory. Without that a validator change is only
  ever exercised against small synthetic fixtures: capping MAX_ROWS at
  400 passes the unit tests and the gate, then breaks the weekly
  automation after merge with "CSV exceeds 400 data rows".

Checking out the base commit also removes last round's git show of the
validator -- at base.sha the reviewed validator is simply the one in the
workspace.

Rehearsed per case, with each branch's diff verified first: a malicious
PR that swaps the CSV from a test now exits 1 where it previously passed
green; the current automation snapshot still validates to the hash
published in #1617; a corrupt CSV, a stricter validator, and a CI-only
change all land where they should.
2026-09-24 09:35:18 +02:00
jack
58ccb30575
Route GeoRelay updates through reviewed data (#1436)
Replaces the scheduled workflow that pushed upstream georelay CSV directly to main unreviewed with a validator-backed automation that resolves an immutable upstream commit, validates against the reviewed baseline, and opens a non-auto-merged PR (or tracking issue) instead of writing to main. Adds a mirrored strict client-side validator in GeoRelayDirectory (all-or-nothing, size/row/host/coord caps, streamed 512KiB cap, >=50% baseline overlap) that fails safe to last-known-good, and retargets runtime refresh to bitchat's reviewed copy.
2026-07-26 10:55:07 +02:00