Skip to content

test(db): pin the fence verdict guard; fix the two #3829-broken owner-limit tests - #4148

Open
tlongwell-block wants to merge 2 commits into
mainfrom
eva/pr4124-followup-tests
Open

test(db): pin the fence verdict guard; fix the two #3829-broken owner-limit tests#4148
tlongwell-block wants to merge 2 commits into
mainfrom
eva/pr4124-followup-tests

Conversation

@tlongwell-block

@tlongwell-block tlongwell-block commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Test-only follow-up from the #4124 review round. No production code changes.

1. Two-entry fence fixture — pins the verdict guard

Dawn's mutation testing on #4124 found that the over-budget leg of is_relay_member_is_bounded_routed_and_fails_closed could not independently pin either staleness guard: with a single injected fence entry, the entry is simultaneously the ring's newest AND the one the reader proves, so the cheap precheck alone fails the read closed — the authoritative per-session verdict guard could be deleted with byte-identical test results. Sami then proved the proposed one-line reason == "stale" assert kills neither mutant (both guards emit the identical label; the precheck short-circuits) and designed this fixture instead; Dawn independently rebuilt it and confirmed.

The new leg expresses the real replication-lag case — a reader session proving an older entry than the shared ring's newest:

  • pin the replica's observable token to 100, then record a proved-but-stale entry (token 50, committed 10s ago) and a fresh unobservable one (token 999, just committed), both under the replica's real heartbeat epoch (a fresh UUID would route to the writer via EpochMismatch — the wrong reason, and the test would pass while proving nothing)
  • the precheck passes on the fresh newest; resolve() proves the stale token-50 entry; only the verdict stage can catch that
  • asserts the outcome (the writer's divergent row is served), the writer/stale route metric, and the absence of any replica decision — label as corroboration, outcome as proof

Mutation results (verified locally at this head):

Mutant Result
Verdict stage: Bounded => Some("fresh") FAILS — killed (was surviving)
Precheck: never Err("stale") passes — correct: it is availability hygiene, not soundness (replica_fence.rs); the verdict stage re-decides authoritatively

Credit: diagnosis Dawn, fixture design + patch Sami (verified on merged main ac4fa13b8), independent reproduction Dawn.

3. Seed the operator owner-limit test from the function, not the constant

Same defect class as item 2, found by Sami after this PR opened: fresh_host_at_owner_limit_returns_limit_reached_conflict (buzz-relay/api/operator.rs) looped over the MAX_COMMUNITIES_PER_OWNER constant while provision_community enforces max_communities_per_owner(), which reads BUZZ_MAX_COMMUNITIES_PER_OWNER with the constant only as fallback. Latent today (nothing wires that env var), but the moment an operator sets it the loop count and the enforced limit diverge and the CONFLICT assertion inverts. Verified RED/GREEN with BUZZ_MAX_COMMUNITIES_PER_OWNER=3: before, 409 at iteration 4 of 5; after, passes at env=3, env=7, and default.

2. Un-break the two owner-limit tests (#3829)

#3829 raised the default per-owner community limit 3 → 5 without updating create_community_with_owner_enforces_per_owner_limit and transfer_ownership_returns_limit_reached_for_maxed_transferee; both have failed on main since (they seed exactly 3, then expect LimitReached). Both now seed from max_communities_per_owner() — the same value the code enforces — so they track the configured limit instead of a literal.

Validation

At 7db75a607 (branched from merged main ac4fa13b8):

  • Full PG-gated buzz-relay suite serial: 37/0; non-PG buzz-relay: 835 passed, 1 failed — mesh_demo::demo_join_forwarded_arm_round_trips_echo, a known pre-existing flake unrelated to this diff (fails identically on the clean previous head in the same environment)
  • Operator limit test RED/GREEN as above

At 8fd5ab04d (previous head, buzz-db portion unchanged since):

  • Full PG-gated buzz-db suite serial: 153 passed / 0 failed (main is 151/2 — the two failures are the feat(relay): raise hosted community limit to five #3829 tests fixed here)
  • Non-PG buzz-db suite: 94/0
  • Verdict mutant applied → routed test fails; reverted → passes (RED/GREEN, same shell)
  • cargo fmt --check + cargo clippy -p buzz-db --all-targets clean

Two test-only changes, both follow-ups from the #4124 review round; no
production code is touched.

1. Two-entry fence fixture (Sami's design, from Dawn's diagnosis): the
   over-budget leg of is_relay_member_is_bounded_routed_and_fails_closed
   used a single injected fence entry, which is simultaneously the ring's
   newest AND the entry the reader proves — so the cheap precheck alone
   fails the read closed, and the authoritative per-session verdict guard
   could be deleted with byte-identical results (mutation-proven). The new
   leg records a proved-but-stale entry (token 50, committed 10s ago) plus
   a fresh unobservable one (token 999) under the replica's real epoch:
   the precheck passes on the newest while resolve() proves the stale one,
   which only the verdict stage can catch. Asserts the boolean outcome
   (writer's row visible), the writer/stale route metric, and the absence
   of any replica decision. Kills the verdict mutant; the precheck mutant
   correctly survives (replica_fence.rs: availability hygiene, not
   soundness).

2. Owner-limit tests seed from max_communities_per_owner() instead of a
   hardcoded 3. #3829 raised the default limit 3 -> 5 without updating
   create_community_with_owner_enforces_per_owner_limit or
   transfer_ownership_returns_limit_reached_for_maxed_transferee; both
   have failed on main since. Seeding from the enforced limit fixes them
   and keeps them limit-change-proof.

Verified: full buzz-db PG-gated suite 153/153 (was 151/2 on main);
verdict mutant now FAILS the routed test; fmt + clippy clean.

Co-authored-by: Tyler <109685178+tlongwell-block@users.noreply.github.com>
Signed-off-by: Tyler <109685178+tlongwell-block@users.noreply.github.com>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner August 1, 2026 17:56
fresh_host_at_owner_limit_returns_limit_reached_conflict looped over the
MAX_COMMUNITIES_PER_OWNER constant while provision_community enforces
max_communities_per_owner(), which reads BUZZ_MAX_COMMUNITIES_PER_OWNER
with the constant only as fallback. Latent today (nothing wires the env),
but the moment an operator sets it the test provisions the wrong count
and the CONFLICT assertion inverts. Same defect class as the two limit
tests fixed in the previous commit: literal-vs-const there, const-vs-env
here. Found by Sami.

Verified RED/GREEN with BUZZ_MAX_COMMUNITIES_PER_OWNER=3: before, the
loop runs 5 while the code enforces 3 (409 at iteration 4); after, the
test passes at env=3, env=7, and default.

Co-authored-by: Tyler <109685178+tlongwell-block@users.noreply.github.com>
Signed-off-by: Tyler <109685178+tlongwell-block@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant