Skip to content
This repository was archived by the owner on Sep 8, 2026. It is now read-only.

contributor-rewards: add golden tests covering reward values - #414

Merged
bgm-malbeclabs merged 5 commits into
mainfrom
test/contributor-rewards-goldens
Aug 27, 2026
Merged

bgm-malbeclabs merged 5 commits into
mainfrom
test/contributor-rewards-goldens

Conversation

@bgm-malbeclabs

Copy link
Copy Markdown
Collaborator

Summary of Changes

  • Adds the first tests in this crate that cover reward values. Before this, contributor-rewards had 113 unit tests and 8 integration tests, and not one asserted a reward figure. This crate decides what contributors are paid.
  • Adds a committed fixture, tests/goldens/mainnet-beta-epoch-129-trimmed.json, plus make-fixture.py that regenerates it from a full capture.
  • Adds two golden tests over that fixture: one pinning the aggregated Shapley output, one pinning per-city output. Both run in 0.35s together.
  • One source change: src/ingestor/types.rs gains an AccessPassStatus::Expired to ExpiredDeprecated compat migration, alongside the migrations already in apply_serviceability_json_compat_migrations. Without it the epoch-129 capture cannot be deserialized at all. Verified against upstream accesspass.rs:112, same discriminant retained. The fixture holds 10 Expired passes, so it is genuinely exercised.
  • CHANGELOG.md updated.

Tracks malbeclabs/infra#2392. Prompted by the monorepo migration design (malbeclabs/doublezero#4239), whose step 4 merges this crate into a shared Cargo workspace. That merge can change resolved dependency versions, and nothing today would notice if reward figures moved as a result.

Why the fixture is a trimmed subnetwork

Two constraints forced this shape, and both were measured rather than assumed.

The existing tests/testnet_snapshot.json produces 0.0 for every operator in every city. That is genuine output, not a harness bug: the Shapley machinery works (evaluator.rs's own test asserts nonzero proportions), the input assembly matches PreparedData::from_snapshot builder for builder, and test_pvt_links.rs pins real latencies from the same snapshot. Those private links are simply no faster than public internet on those routes, so no operator adds value. Pinning zeros would give a test that passes forever and detects nothing.

The full mainnet capture goes the other way. Exact Shapley is O(2^n) in operators, and 14 operators across 30 cities was killed after 49 CPU-minutes without finishing.

So the fixture keeps 4 operators and 6 cities, which produces nonzero rewards in 0.3 seconds. The coverage limit is real and worth stating: a regression reachable only through the dropped topology goes unseen. That is a much smaller gap than the zero coverage it replaces.

Why values use a tolerance rather than exact equality

Structure is asserted exactly: operator set, operator ordering, city set, per-city ordering, all counts. Values and proportions use a 1e-12 relative tolerance.

The computation is deterministic on one machine, verified under RAYON_NUM_THREADS of 1, 2 and 16 and bit-identical in a release build with different codegen and LTO. Bit-identical floating point across architectures is not guaranteed, though, and these goldens were generated on arm64 while CI is x86_64. An exact gate would go permanently red the day CI changes architecture, and a permanently red test is one everybody learns to ignore.

assert_close also skips entries where both sides are below 1e-6. Seven of the 24 per-city entries are cancellation noise near 1e-12, against a largest real value of 2331.52. Gating those at an absolute 1e-12 would fail on differences that carry no reward information, and the natural response to that red is UPDATE_GOLDEN=1, which also quietly destroys the detector. The clamp requires both sides to be negligible, so a real value collapsing to zero is still caught.

Testing Verification

  • Both goldens fail when perturbed, and pass again after restore. Verified three ways: shifting a large value in each golden, replacing a large value with zero, and restoring to a byte-identical file with a clean git status. A golden test that cannot fail is the failure mode that matters here, and passing twice does not prove it can.
  • Sensitivity is roughly 1:1. Perturbing contiguity_bonus by 2e-10 moves the output by 2.5e-10, so a real change in reward math lands 8 or more orders of magnitude above the 1e-12 gate.
  • Determinism confirmed across RAYON_NUM_THREADS 1, 2 and 16, and between debug and release builds.
  • The compat migration was checked against the vendored doublezero-serviceability source rather than inferred from the error.
  • Full crate suite green: 13 test targets, zero failures. just fmt-check and just clippy clean.
  • Runtime 0.35s debug, 0.06s release.

Known follow-ups, deliberately not in this PR

  • make-fixture.py truncates sample arrays but does not filter telemetry records to the surviving devices, so 254 of 328 device records are inert. Proven safe to leave: a filtered copy (4.66 MB to 3.33 MB) reproduces the golden exactly, so the filter can be added later with no regeneration. Noted in the script's docstring.
  • malbeclabs/infra#2393 proposes a [profile.test] opt-level for the workspace. Not needed for these tests at 0.3s, but the two Rust CI jobs currently take 17 and 20 minutes.

Add a trimmed mainnet-beta snapshot fixture and the script that produces it, for a
future Shapley golden test. The only previously committed snapshot (testnet) produces
zero reward for every operator, so no golden test can assert a real value against it.

Exact Shapley computation is O(2^n) in the operator count, so the fixture keeps only
the contributors and cities that dominate device count on the full mainnet-beta
capture, which keeps the sub-network connected and the computation fast. It is not a
byte-size trim of the full topology.

Also migrate the access pass status value Expired to ExpiredDeprecated when loading a
snapshot captured before doublezero-serviceability PR #3831 renamed that enum variant,
without which the full mainnet-beta capture fails to deserialize.
Clamp near-zero Shapley entries in assert_close so cancellation noise around
1e-12 does not sit below its own comparison gate. Fix UPDATE_GOLDEN to
regenerate only on the value "1", not any set value. Rename the mn-beta
fixture and goldens to mainnet-beta, fix the wrong regeneration path in
make-fixture.py, and apply the remaining minor cleanups from the review.
Comment thread crates/contributor-rewards/src/ingestor/types.rs Outdated
Comment thread crates/contributor-rewards/CHANGELOG.md Outdated
@bgm-malbeclabs
bgm-malbeclabs enabled auto-merge (squash) August 27, 2026 21:24
@bgm-malbeclabs
bgm-malbeclabs merged commit ceeee22 into main Aug 27, 2026
6 checks passed
@bgm-malbeclabs
bgm-malbeclabs deleted the test/contributor-rewards-goldens branch August 27, 2026 22:02
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants