Skip to content

fix: deflake should_flag_only_deposits_at_or_above_the_per_token_minimum - #11408

Open
basvandijk wants to merge 1 commit into
masterfrom
ai/deflake-deposit_from_cex-2026-09-01
Open

fix: deflake should_flag_only_deposits_at_or_above_the_per_token_minimum#11408
basvandijk wants to merge 1 commit into
masterfrom
ai/deflake-deposit_from_cex-2026-09-01

Conversation

@basvandijk

Copy link
Copy Markdown
Collaborator

Problem

//rs/ethereum/cketh/minter:deposit_from_cex kept flaking after #11383 (which removed the dominant stop-canister race). Both post-#11383 master flakes share one new signature: should_flag_only_deposits_at_or_above_the_per_token_minimum fails at its first assertion with

assertion failed: `Scanning { valid_until: …, last_scanned_block: Some(Nat(11)), scan_count: 1 }`
does not match `DepositStatus::AwaitingSweep(detected) if …`

right after credit_deposits funded the deposit addresses on anvil (its own balance assertions passed, so the funds were readable).

Root cause

The minter's balance scan does not fetch a block number: it eth_calls at its cached latest_block_height (rs/ethereum/cketh/minter/src/balance_scan/mod.rs, pinned as BlockTag::Number), and that cache is refreshed only by the separate 30-second REFRESH_LATEST_BLOCK_HEIGHT_INTERVAL timer (deposit.rs is its sole writer; the log scrape writes a different field). So a scan firing after the harness mined the funding block can still read every balance at a pre-funding block: the call succeeds (the token code, set through anvil_setCode, is visible to historical eth_calls — mined transfers are not), finds nothing, and record_scan stores scan_count = 1 with the stale last_scanned_block.

The failing logs show exactly this: an ETH log scrape covering blocks 1..=11 (head ≥ 11 pre-funding), the funding mined after it, then one balance scan — scanned 3 (address, token) pair(s), found 0 candidate(s), 0 decode error(s), 0 call error(s) — pinned at block 11.

The harness' await_scan accepted any Scanning status with scan_count >= 1 as terminal, so it returned that transient verdict instead of driving on until a scan that could actually see the funds — and the test's AwaitingSweep assertion failed. (The next scan corrects the verdict within a couple of blocks; the predicate just never waited for it.)

Fix

await_scan now snapshots the anvil chain head at entry (the funding is always mined by then — every caller funds through credit_deposits first) and counts an address as scanned only once its status is:

  • AwaitingSweep — terminal by construction (a detection only results from a scan that read a balance at or above the minimum, and a queued pair never returns to Scanning), or
  • Scanning with scan_count >= 1 and last_scanned_block at or past that entry head — a scan pinned there cannot have missed the balance, so a below-minimum verdict is genuine.

SCAN_TICKS grows from 4 to 8 (a budget, not a cost — driving stops the moment the condition holds): the pair's block-based backoff can demand up to 300 block-seconds before it is due for the scan the new predicate waits for, against the ~10 blocks each tick's settle mines.

Validation

  • Deterministic reproduction: temporarily deploying the token code before registration (so a pre-funding scan completes, recording exactly the state the stale-cache race records — scan_count = 1 at a block where the funding is invisible) and forcing one such scan:
    • old predicate → fails with the exact CI signature, deterministically;
    • fixed predicate → converges to AwaitingSweep and the test passes end to end.
  • cargo check, cargo fmt, ./ci/scripts/rust-lint.sh — clean.
  • Depth-2 bazel rdeps all pass: integration_tests_tests/{ckerc20,cketh}_test, minter:lib_tests, test_utils:lib_tests.
  • bazel test --runs_per_test=3 --jobs=3 //rs/ethereum/cketh/minter:deposit_from_cex — 3/3 full-suite runs pass.

The remaining non-success runs in the window since #11383 merged all ran on commits that predate it (branch/merge-queue checkouts from before the merge) and show the old, already-fixed stop-canister signature.

This PR was created following the steps in .claude/skills/fix-flaky-tests/SKILL.md.

🤖 Generated with Claude Code

The minter's balance scan reads every balance at its cached
latest_block_height, refreshed only by a separate 30-second timer - so
a scan can run after the harness funded the deposit addresses on anvil
and still eth_call at a pre-funding block, recording scan_count = 1
without having seen the funds. The live harness' await_scan accepted
any Scanning status with scan_count >= 1 as terminal and returned that
transient verdict, so the test's AwaitingSweep assertion flaked.

An address now counts as scanned only once its status is AwaitingSweep
(terminal by construction) or Scanning with a last_scanned_block at or
past the chain head as of await_scan entry - the funding is mined by
then, so such a scan cannot have missed the balance. SCAN_TICKS grows
from 4 to 8 (a budget, not a cost): the pair's block-based backoff gap
can demand up to 300 block-seconds before it is due for the scan the
new predicate waits for.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Deflakes the CEX deposit test by ensuring balance scans include the funded block.

Changes:

  • Requires scans to reach the Anvil head captured when waiting begins.
  • Increases the scan-tick budget to accommodate backoff.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@basvandijk
basvandijk marked this pull request as ready for review September 1, 2026 21:05
@basvandijk
basvandijk requested a review from a team as a code owner September 1, 2026 21:05
@github-actions github-actions Bot added the @defi label Sep 1, 2026
@zeropath-ai

zeropath-ai Bot commented Sep 1, 2026

Copy link
Copy Markdown

No security or compliance issues detected. Reviewed everything up to 04e0051.

Security Overview
Detected Code Changes
Change Type Relevant files
Enhancement ► rs/ethereum/cketh/test_utils/src/live.rs
      Modify SCAN_TICKS constant and augment await_scan doc comment and logic to account for block visibility and pinned-block requirements in balance scans
► rs/ethereum/cketh/test_utils/src/live.rs
      Update await_scan implementation to consider last_scanned_block and funded_by conditions

@basvandijk basvandijk changed the title fix: deflake //rs/ethereum/cketh/minter:deposit_from_cex fix: deflake should_flag_only_deposits_at_or_above_the_per_token_minimum Sep 1, 2026

@mbjorkqvist mbjorkqvist left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @basvandijk!

@basvandijk
basvandijk added this pull request to the merge queue Sep 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants