Skip to content

fix(test): serialise optimizer_bench fixture writes (flaky recall_file/no_match on main) - #43

Merged
HackPoint merged 1 commit into
mainfrom
agents/t0-bench-fixture-race
Oct 11, 2026
Merged

HackPoint merged 1 commit into
mainfrom
agents/t0-bench-fixture-race

Conversation

@HackPoint

Copy link
Copy Markdown
Owner

Cause

measure() in optimizer_bench.rs rewrites the shared target/bench-fixtures/<name> with fs::write (truncate, then fill) and then lets the tool read it. Two tests in the file call measure() on parallel threads, so one thread's truncate can land under the other's read. The tool sees an empty file, reports full=0, falls back to would_inflate. That is the red job: https://github.com/HackPoint/lumen/actions/runs/38170443513 (recall_file/no_match, full=0 returned=25 vs baseline 7495/230).

Hypothesis confirmed. Throwaway repro on unfixed code (a thread rewriting big.rs in a loop while the main thread ran recall_file/no_match 60 times): 1, 2 and 5 bad runs of 60 over three runs, every one exactly full_tokens: 0, returned_tokens: 25, route: would_inflate.

Fix

A process-wide Mutex held by measure() from the write to the last tool call (poison ignored). The path stays shared because it is part of the reply header and so of the baselined token counts. A rename-over-file swap would fail on Windows while the other thread has the file open. Baseline file and assertions untouched, no retries, no new dependency.

Red, then green

New test measure_calls_never_share_the_fixture_window. It keeps one measure() inside its write-to-read window (via a scenario args hook) and waits up to 400 ms for a second one to walk in. No race to win. Unfixed (3 of 3 runs, 0.37 s):

assertion `left == right` failed: two measure() calls were inside the fixture write-to-read window at once; the file one of them reads is being truncated by the other
  left: 2
 right: 1

Fixed: passes in 0.96 s. The whole bench binary passed 8 of 8 consecutive runs.

Audit (crates//tests, #[cfg(test)] in crates//src, src-tauri)

No other instance of parallel tests writing then reading one shared path. escape_valve, hook_cli, repeat_reads, efficiency, protocol, ranked_metering, e2e_* and the daemon/stats/core tests use per-test tempdirs, and the hook harnesses override TMPDIR per test. Suspected, not fixed:

  • LUMEN_BENCH_BLESS=1 (manual only): the baseline test rewrites optimizer_baseline.json while the_fixtures_match_… reads it. The two accesses are seconds apart.
  • report.rs (GH_TOKEN), update.rs (LUMEN_UPDATE_CHECK), faults.rs (LUMEN_FAULT_SPOOL) and daemon (LUMEN_LOG) tests mutate process env. I checked that nothing else in those binaries reads the vars today.
  • report.rs stages long issue bodies at a fixed temp_dir()/lumen-issue-body.md. No test reaches that branch.

Local gate

/tmp/lumen-verify.sh on this HEAD: GATE GREEN (660 s). Includes silent_fallbacks, fmt and clippy -D warnings, MSRV, hook e2e and the verify-install legs.

…ead never sees a truncated file

measure() rewrites target/bench-fixtures/<name> with fs::write (truncate, then fill) and then
lets the tool read it. Two tests in the file call measure() on parallel threads, so one
thread's truncate could land under the other's read: the tool saw an empty file, reported
full=0 and fell back to would_inflate. That failed
every_mcp_tool_is_measured_against_the_committed_baseline on macos-14 (recall_file/no_match,
full=0 returned=25 against 7495/230).

measure() now holds a process-wide mutex from the write to the last tool call. The path stays
the shared one because it is part of the reply header and so of the baselined token counts.
A new test keeps one measure() inside its write-to-read window and fails if a second walks in.
@HackPoint
HackPoint merged commit 3008af7 into main Oct 11, 2026
17 checks passed
@HackPoint
HackPoint deleted the agents/t0-bench-fixture-race branch October 11, 2026 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant