Skip to content

test(cketh): adversarial end-to-end coverage of sweeper fee funding - #11097

Open
mbjorkqvist wants to merge 131 commits into
masterfrom
mathias/DEFI-2933-hardening
Open

test(cketh): adversarial end-to-end coverage of sweeper fee funding#11097
mbjorkqvist wants to merge 131 commits into
masterfrom
mathias/DEFI-2933-hardening

Conversation

@mbjorkqvist

@mbjorkqvist mbjorkqvist commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Part of DEFI-2933 (sweeper fee funding), the last PR of the stack. Every PR below it has merged, so this one targets master.

Why

The stack's existing live tests prove funding works. This one proves it fails safe. Each test drives a distinct way funding can go wrong through the real pipeline — real ckETH ledger, real EVM RPC canister, local anvil, threshold-ECDSA signature — and asserts the invariant the feature exists to protect: ETH never moves ahead of the ckETH burned to pay for it.

What

Three adversarial scenarios:

  • An empty fee account. The burn fails, so no ETH may move. Held across several withdrawal-timer ticks, since "the minter must not do this" can only be established as a bounded negative — and the test first waits for the log line proving the check ran and declined, so it cannot pass because nothing happened.
  • A sweeper already above the low-water mark. No funding is due, so nothing is burned. Arranged by letting a real funding land and finalize, which is the only thing that moves the bound the decision reads — putting ETH at the address behind the minter's back deliberately would not.
  • A funding transaction that fails on chain. The destination is given code, so a bare transfer runs out of gas: all 21'000 gas it carries is intrinsic, leaving nothing to execute with. The ckETH stays burned and is not reimbursed: no ETH reaches the sweeper, the failed transaction still pays its gas, and the burn minus that gas stays as backing — which is what keeps burned ahead of spent. The test waits for finalization, pins spend against the burn numerically, and then buys a tick longer than the reimbursement interval so that "never reimbursed" is not just true at the moment of finalization.

No live fee-spike test: that ceiling is pinned exactly by unit tests, and reproducing it here would mean driving anvil's base fee up over several ticks for little extra signal.

What it found

Waiting for finalization — rather than asserting mid-flight, as the earlier live test did — surfaced a production defect that would trap the withdrawal timer permanently. The fix shipped in #11086, where it belongs; this PR is what made it visible.

Harness work

These tests need to arrange a chain and then read the minter's own view of it, so the harness gains a fixture that leaves the fee account empty, wrappers over anvil's setCode/getCode, a reader for the minter's /metrics endpoint, a wait on a log line, and a wait for a funding to finalize whatever its outcome — the one already there insists on success, which is the opposite of what the failure test needs.

They join the live tests already in the deposit_from_cex target, which runs one test at a time and is already sized large; the three added here cost roughly a minute each, and the whole target now takes about four minutes.

Stack

# PR Status
#11060 — Read a native ETH balance via the EVM RPC canister merged
#11065 — Burn ckETH from the minter's own fee subaccount closed; folded into #11072 and #11086
#11072 — Add the SweeperFunding withdrawal-request variant merged
#11083 — Burn-first accounting for sweeper fee funding merged
#11086 — Sweeper fee-funding task, with an end-to-end test merged
#11094 — Sweeper funding observability merged
1 Adversarial end-to-end coverage of sweeper fee funding this PR

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds adversarial end-to-end coverage ensuring sweeper fee funding fails safely.

Changes:

  • Tests empty fee accounts, sufficient sweeper balances, and reverted funding transactions.
  • Extends the live harness with dashboard, balance, code, status, and finalization helpers.
  • Adds a dedicated long-running Bazel target.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
rs/ethereum/cketh/test_utils/src/sweeper_funding.rs Expands live-test utilities.
rs/ethereum/cketh/minter/tests/sweeper_funding.rs Uses dashboard-derived sweeper address.
rs/ethereum/cketh/minter/tests/sweeper_funding_hardening.rs Adds three adversarial scenarios.
rs/ethereum/cketh/minter/BUILD.bazel Registers the long-running test target.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread rs/ethereum/cketh/test_utils/src/sweeper_funding.rs Outdated
Comment on lines +148 to +150
let burn_index = setup
.in_flight_funding_burn_index()
.expect("the funding must be in flight once its burn has happened");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and well spotted — fixed in 2096a99.

The ordering is as you describe: fund_sweeper_address awaits the ledger burn and records AcceptedSweeperFundingRequest only after that call returns, while await_burn watches the ledger's total supply through an external query. So there is a real window in which the supply has dropped and the dashboard still shows no in-flight funding, and reading the index there would panic on an empty row.

It now polls, with the deadline as a named constant because it has to sit between two bounds rather than merely being long enough: generous for an inter-canister hop, but well under the roughly six minutes before the transaction finalizes and clears that row again. Polling indefinitely would have traded this race for the opposite one.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

rs/ethereum/cketh/minter/tests/sweeper_funding_hardening.rs:83

  • The no-burn baseline is captured after upgrade_minter(), which starts the zero-delay funding timer on this live PocketIC instance. The task can therefore complete its ledger burn before these queries run, causing both “unchanged” assertions to compare against an already-debited state. Capture the supply and fee-account balance before re-arming the timer so the test cannot miss the behavior it is intended to reject.
    let supply_before = setup.cketh_total_supply();
    let fee_account_before = setup.cketh_balance_of(setup.fee_account());

// zero-delay timer, so a minter that wrongly funded could burn before these queries returned
// and both assertions below would then compare against an already-debited state — passing
// precisely when the behaviour they reject had happened.
let supply_before = setup.cketh_total_supply();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replying to the suppressed comment about this baseline being captured after upgrade_minter() (no comment node to thread onto, hence a fresh one here).

Right, and a fair catch — fixed in 532989e.

Worth being precise about the failure mode, because it is not a flake in the current code: with a correct minter the sweeper is above the low-water mark, no funding is due, and nothing is burned, so the ordering never bites. It matters for a broken minter — if the post-upgrade check wrongly funded, its burn could land before those two queries returned, and then both "unchanged" assertions would compare against an already-debited state. The test would go green at precisely the moment it was supposed to fail.

The baselines are now read before upgrade_minter(), which is the one point where nothing can yet have happened: the fee account is funded, the sweeper is arranged, and the next scheduled check is a whole interval away. The revert test below already did it in that order — I fixed it there when addressing the in-flight race and did not carry the reasoning back up.

I left the first test as it is: its fee account is empty, so no burn can succeed at all, and its log assertion proves the task ran and declined at the burn rather than never running.

Rachit2323 pushed a commit to Rachit2323/ic that referenced this pull request Aug 12, 2026
…ity#11060)

Part of [DEFI-2933](https://dfinity.atlassian.net/browse/DEFI-2933)
(sweeper fee funding), first of a seven-PR stack.

## Why

Funding the sweeper address with gas requires knowing how much gas it
already holds. The EVM RPC canister exposes no endpoint for a native ETH
balance, and its Rust client offers no getter for one, so the minter
currently has no way to ask.

## What

Reads the balance through the EVM RPC canister's generic JSON-RPC
passthrough, which forwards a payload to every provider and agrees on
one answer under the configured consensus strategy. That strategy is a
threshold of the providers — 3 of 4 on mainnet, 2 of 4 on Sepolia — and
it is the only agreement accepted: there is no client-side reduction, so
a result the canister reports as inconsistent stays an error rather than
being resolved by picking a winner.

Because the canister deserializes each response's `result` field, what
the minter receives is the quantity itself rather than any surrounding
JSON. It is therefore decoded exactly: quotes, padding, leading zeros
and sign characters are the provider's own malformation and are rejected
rather than repaired.

A failed read is an error, never a zero. This is the decision the rest
of the stack depends on: confusing "could not read the balance" with "no
gas left" would burn ckETH to top up an address that is already funded,
which is pure loss. The request builder and the result decoder are pure
functions so both sides of that guarantee are pinned directly, including
a test asserting that no error input can decode to a zero balance.

The route was also proven end to end against a live EVM RPC canister and
a local anvil node, reaching 3-of-4 consensus at both `latest` and
`finalized`.

## Stack

Merge in order; each PR targets the one above it.

| # | PR | Status |
|---|----|--------|
| 1 | Read a native ETH balance via the EVM RPC canister | **this PR** |
| 2 | dfinity#11065 — Burn ckETH from the minter's own fee subaccount | ready
for review |
| 3 | dfinity#11072 — Add the SweeperFunding withdrawal-request variant | ready
for review |
| 4 | dfinity#11083 — Burn-first accounting for sweeper fee funding | ready for
review |
| 5 | dfinity#11086 — Sweeper fee-funding task, with an end-to-end test |
Copilot re-review pending, CI green incl. long tests |
| 6 | dfinity#11094 — Sweeper funding observability and the prepaid-gas gate |
open |
| 7 | dfinity#11097 — Adversarial end-to-end coverage of sweeper fee funding |
open |

[DEFI-2933]:
https://dfinity.atlassian.net/browse/DEFI-2933?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mbjorkqvist
mbjorkqvist force-pushed the mathias/DEFI-2933-observability branch from 4d829c1 to e5adda2 Compare August 14, 2026 14:32
@mbjorkqvist
mbjorkqvist force-pushed the mathias/DEFI-2933-hardening branch from 532989e to b48a599 Compare August 14, 2026 14:54
@mbjorkqvist
mbjorkqvist force-pushed the mathias/DEFI-2933-observability branch from b633db9 to 9d7cb6d Compare August 18, 2026 16:17
@mbjorkqvist
mbjorkqvist force-pushed the mathias/DEFI-2933-hardening branch from b48a599 to df115ab Compare August 18, 2026 16:28
@mbjorkqvist
mbjorkqvist force-pushed the mathias/DEFI-2933-observability branch from 9d7cb6d to c062a60 Compare August 18, 2026 17:54
@mbjorkqvist
mbjorkqvist force-pushed the mathias/DEFI-2933-hardening branch from df115ab to fc087d2 Compare August 18, 2026 17:54
@mbjorkqvist
mbjorkqvist force-pushed the mathias/DEFI-2933-observability branch from c062a60 to 5487cb6 Compare August 19, 2026 07:18
@mbjorkqvist
mbjorkqvist force-pushed the mathias/DEFI-2933-hardening branch from fc087d2 to c4eefd7 Compare August 19, 2026 07:29
@mbjorkqvist
mbjorkqvist requested a balanced review from Copilot August 19, 2026 07:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

rs/ethereum/cketh/minter/tests/sweeper_funding_hardening.rs:196

  • Checking only that the surplus is nonzero does not verify the stated failed-transaction accounting. If finalization accidentally recorded a zero transaction fee, the entire burn would appear as surplus and this test would still pass. Assert that sweeper-eth-spent is nonzero after finalization so the end-to-end path proves failed transactions still charge and record gas.
    assert_ne!(
        surplus, "0 Wei",
        "the unreimbursed burn must still show as burned but unspent, got {surplus}"
    );

rs/ethereum/cketh/minter/tests/sweeper_funding_hardening.rs:123

  • This description is incorrect for a failed Ethereum transaction: the transfer value is reverted, but the sender still loses the effective gas fee. Clarify that only the value fails to reach the sweeper; otherwise this contradicts the accounting exercised below.

This issue also appears on line 193 of the same file.

/// A funding transaction that fails on chain is never reimbursed. The ETH never leaves the main
/// address and the ckETH stays burned, so the burn ends up as extra backing.

pull Bot pushed a commit to bit-cook/ic that referenced this pull request Aug 19, 2026
…ty#11072)

Part of [DEFI-2933](https://dfinity.atlassian.net/browse/DEFI-2933)
(sweeper fee funding). Now targets `master`: dfinity#11060 has merged, and
dfinity#11065 is being closed with its contents folded into the PRs that use
them — the fee-subaccount constant into this one, the burn helper into
dfinity#11086 alongside its caller.

## Why

Sweeper fee funding is mechanically an ordinary ckETH withdrawal — same
nonce sequence, same threshold-ECDSA signing, same fee-bumped
resubmission — so it becomes a third `WithdrawalRequest` variant rather
than a parallel pipeline.

It differs in exactly one respect, and that difference is what the whole
feature turns on: the ckETH burned for funding is **never** re-minted. A
funding request must therefore never reach the reimbursement machinery.

## What

Three places enforce that, all of which would otherwise fail only at
runtime:

- `maybe_reimburse` is the double-minting guard, and
`record_reimbursement_request` asserts membership has been cleared
before minting. Funding is kept out of the set on insert, and the
corresponding assertion on removal is made conditional. Both are
production assertions, so a missed branch traps the canister.
- The conversion to a reimbursement index becomes fallible,
deliberately: a fallible conversion makes the compiler prove at every
call site that funding cannot produce an index, rather than relying on a
panicking arm that traps if a site is missed. The two callers construct
their index inside the reimbursable arms instead.
- Nothing is recorded to pay back if a funding transaction finalizes
with a failure receipt — an outcome that requires code at the
destination, so it cannot arise for a bare transfer to the sweeper, and
is logged as unexpected rather than handled. The ckETH stays burned
while no ETH reaches the sweeper — only the gas the failed transaction
still paid leaves the main address — so the rest of the burn over-backs
ckETH instead of anything being lost, and neither a reimbursement path
nor a second event type is needed.

Everything else follows ckETH: the 21'000 gas limit of a plain value
transfer to a code-less address, a resubmission strategy ceilinged at
the burned amount — so a climbing gas price shrinks the ETH delivered to
the sweeper rather than spending more than was burned — and a fee carved
out of that same amount, so balance accounting needs no change.

Since reimbursement is the only difference, the variant carries an
`EthWithdrawalRequest` — the same payload a user's ckETH withdrawal
carries — rather than a near-copy of it, and the spec drops the one
piece of funding accounting that had no counterpart in the withdrawal
flow: burned-but-unspent amounts are no longer credited against the next
funding's burn. They stay as backing, just like the unspent gas a user's
withdrawal leaves behind.

Funding appears in the withdrawal status endpoint, the dashboard and the
event log rather than being hidden: it moves ckETH-denominated value and
is a public, auditable action. A user query never matches one, since the
sender is the minter itself.

The new event takes tag 27; 26 went to `AutomaticDepositReceived` on
master while this branch was open, and the tags are the durable CBOR
encoding, so they cannot collide.

## Stack

Merge in order; each PR targets the one above it.

| # | PR | Status |
|---|----|--------|
| — | dfinity#11060 — Read a native ETH balance via the EVM RPC canister |
merged |
| — | dfinity#11065 — Burn ckETH from the minter's own fee subaccount | closed;
folded into dfinity#11072 and dfinity#11086 |
| 1 | Add the SweeperFunding withdrawal-request variant | **this PR** |
| 2 | dfinity#11083 — Burn-first accounting for sweeper fee funding | ready for
review |
| 3 | dfinity#11086 — Sweeper fee-funding task, with an end-to-end test | draft
|
| 4 | dfinity#11094 — Sweeper funding observability and the prepaid-gas gate |
draft |
| 5 | dfinity#11097 — Adversarial end-to-end coverage of sweeper fee funding |
draft |

[DEFI-2933]:
https://dfinity.atlassian.net/browse/DEFI-2933?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mbjorkqvist
mbjorkqvist force-pushed the mathias/DEFI-2933-observability branch from 5487cb6 to 572e5a0 Compare August 19, 2026 08:42
The failure recording was a two-call protocol — bump the counter, then
record a zeroed transfer — with nothing tying the two together and a test
comment documenting the ordering as a convention. The outcome is now an
argument to `record_finalized_funding`, so the count and the undelivered
transfer are one decision, and `record_failed_funding` stops being API. The
sweeper block also leaves `update_balance_upon_withdrawal`, whose name says
nothing about fundings, for a method of its own.

The age gauge emitted NaN when a funding's acceptance time was unknown, which
in Prometheus makes `age > threshold` false — the wedged funding the metric
exists for would never have alerted. The branch is gone (it was unreachable
anyway: the funding task always sets the time), and both ages now read from
the `now_nanos` already bound above.

`burned_not_yet_spent` saturates, since the metrics and the dashboard read it
from a query and an under-backed minter is exactly when those surfaces need
to be up. The transition that could cause it still traps, through a checked
helper.

Metrics: one prefix for all of them, `_total` kept where Prometheus wants it,
HELP trimmed to what each one measures, and the failure count replaced by
`finalized_total{status}` so a failure rate and the funding throughput are
both expressible. The low-water mark and target are exported too, so "the
sweeper needs gas and is not getting it" can be alerted on as
`gas_balance < low_water_mark` without hardcoding a threshold that moves with
the minimum withdrawal amount.

Dashboard: units in the header and bare numbers in the cells, as the rest of
the page does; `Address` instead of a stringly-typed field; one `match`
idiom; the config bound once; the address row named for the gas-paying
address it is, so it cannot be read as the sweeper contract; and a funding in
the withdrawal table marked as one rather than appearing as an unexplained
ckETH withdrawal.

Tests: one exact in-flight assertion in place of two substring ones, which
also pins the timestamp rendering; the sweeper tests moved into page order;
and a `/metrics` test asserting all six are exported, which is the only thing
that catches a mistyped name before Grafana silently shows nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pull Bot pushed a commit to bit-cook/ic that referenced this pull request Aug 26, 2026
…ty#11086)

Part of [DEFI-2933](https://dfinity.atlassian.net/browse/DEFI-2933)
(sweeper fee funding), first of the three remaining PRs. Targets
`master`, now that dfinity#11083 has merged.

Also carries the fee-subaccount burn helper, folded in from dfinity#11065 —
that PR was closed rather than merged, since on its own it was a
constant and a function nothing called. It lands here instead, next to
its only caller and the end-to-end test that drives it through the
minter. The ICRC-2 rule it depends on — that a spender may spend from an
account only when it names that account's own subaccount — is covered
for every ledger in dfinity#11139.

## What

The periodic task that keeps the dedicated sweeper address funded:
derive the address, decide whether a top-up is due, burn ckETH from the
minter's fee subaccount, and only then queue the transfer. Everything
after the burn is the existing withdrawal pipeline. A failed burn halts
funding — and eventually sweeping — rather than moving ETH nothing has
paid for.

## How it knows the sweeper is low

Not by reading the chain. The minter already records what it sent: the
ETH that finalized fundings delivered is a **lower bound** on the
sweeper's balance, and nothing else debits that address until sweeping
lands (S2), at which point the bound subtracts the gas submitted sweeps
provisioned. ETH sent there by anyone else only pushes the true balance
above the bound.

A bound is the right input for the decision, because it errs in the safe
direction: at or above the low-water mark it proves no funding is
needed; below it, we may top up when we need not, which costs a burn we
are backing anyway and self-corrects, since our own transfer raises the
bound. It also means the funding task makes no HTTPS outcall at all — no
provider agreement to choose, no block height to pick, no
unreadable-balance path to handle, and nothing to go stale.

## What a funding burns

The amount it moves, and nothing else: the fee is carved out of that
same amount, exactly as for a user withdrawal, so a top-up of 0.3 ETH —
the target ten minimum withdrawals imply on mainnet — burns 0.3 ckETH
and the sweeper receives 0.3 minus the fee. Whatever part of the
provisioned fee goes unpaid stays as backing rather than being credited
against the next funding — the simplification agreed for this iteration,
which leaves funding accounted for the same way withdrawals already are.

The amount always clears the ledger's own minimum without a floor of its
own, since a funding moves at least the configured headroom and
validation keeps that at or above the minimum withdrawal amount. A real
burn is what gives the funding the ledger index the whole withdrawal
pipeline is keyed by.

Only one funding is allowed in flight at a time, which the planner reads
off the withdrawal pipeline — a funding request that is pending, or has
a created or sent transaction, is outstanding. Deriving it rather than
mirroring it in the accounting means the two cannot disagree, so there
is no inconsistent state to detect. The rule itself is prudence rather
than a correctness requirement — two fundings would each be covered by
their own burn — but it keeps a single funding on the withdrawal nonce
lane and the accounting easy to follow.

## A production guard the adversarial test forced

Finalizing a funding debits the minter's ETH balance counter, which is
credited only by deposits. In steady state every debit is covered by its
own burn, so the counter cannot go negative — but on a fresh deployment
it starts at zero while the main address may already hold ETH, and
nothing previously stopped a funding from reaching for it. The debit at
finalization would then underflow and trap, in the withdrawal timer,
which would be stuck permanently and head-of-line block every user
withdrawal behind it. It does not need anything to go wrong: a perfectly
successful funding triggers it.

Planning now refuses to fund beyond the deposit-backed balance and logs
why. Waiting is the safe direction — the sweeper simply stays unfunded
until deposits cover it.

This was invisible until the adversarial test in the last PR of the
stack waited for finalization; both live tests previously finished while
the transfer was still in flight, which is also why the assertions could
not see it.

## Live tests that no longer wait

The funding path sits behind the 6-minute withdrawal timer, so the
end-to-end test used to spend that time on the wall clock. It now buys
the ticks with `advance_time`, which works on a live instance:
auto-progress sets the time once and thereafter advances *by* elapsed
deltas, so an explicit jump sticks. **372s to 18.6s**, and the target is
no longer tagged `long_test`, so it runs in the ordinary pipeline. The
harness also sets the certified time before going live, the same race
dfinity#11299 just fixed for the balance-scan harness.

Two rules the harness documents at the constants that enforce them: a
jump fires a due interval timer *once* however far it jumps, so N ticks
need N jumps; and nothing may be in flight when time moves, because
`CANISTER_HTTP_TIMEOUT_INTERVAL` is 60 seconds and PocketIC uses the
real payload builder — so the harness lets outcalls drain after each
jump.

## Cost

Replaying the recorded mainnet event log through the funding accounting
is measured at 1.177B instructions in `bench_post_upgrade`, against
1.187B on `master` — so it costs nothing measurable next to what
master's own changes have moved in the meantime. `canbench/results.yml`
records the figure, regenerated after each merge rather than reconciled
by hand.

## Stack

Merge in order; each PR targets the one above it.

| # | PR | Status |
|---|----|--------|
| — | dfinity#11060 — Read a native ETH balance via the EVM RPC canister |
merged |
| — | dfinity#11065 — Burn ckETH from the minter's own fee subaccount | closed;
folded into dfinity#11072 and dfinity#11086 |
| — | dfinity#11072 — Add the SweeperFunding withdrawal-request variant |
merged |
| — | dfinity#11083 — Burn-first accounting for sweeper fee funding | merged |
| 1 | Sweeper fee-funding task, with an end-to-end test | **this PR** |
| 2 | dfinity#11094 — Sweeper funding observability and the prepaid-gas gate |
draft |
| 3 | dfinity#11097 — Adversarial end-to-end coverage of sweeper fee funding |
draft |

[DEFI-2933]:
https://dfinity.atlassian.net/browse/DEFI-2933?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: IDX GitHub Automation <infra+github-automation@dfinity.org>
mbjorkqvist and others added 9 commits August 27, 2026 09:33
Copilot is right that a funding which failed on chain leaves its whole
undelivered amount in this gauge, not just an unpaid fee, so the jump an
operator sees beside a failure was unexplained.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Also follows the metric rename through: `SWEEPER_GAS_BALANCE` still named
`cketh_minter_sweeper_gas_balance`, which no longer exists. The constant is a
string, so nothing failed to compile — only the live test would have, on a
lookup that finds nothing.
…e none

`dashboard.rs`, its tests, `encode_metrics`, the new `State` method and the
metrics test all sit among items that carry no comments at all, so the ones
added here stood out rather than helped. The metric HELP strings stay: those
are what an operator reads.

Kept in `state/sweeper_funding.rs` and `sweeper/tests.rs`, where every
neighbour is documented, but trimmed to what the code does not already say.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Regenerate the minter's canbench baseline. Master's ckETH changes and this
branch's each stay under the 2% noise threshold alone, but they add up to
+2.01% in the merge.
Base automatically changed from mathias/DEFI-2933-observability to master September 3, 2026 07:30
pull Bot pushed a commit to bit-cook/ic that referenced this pull request Sep 3, 2026
Part of [DEFI-2933](https://dfinity.atlassian.net/browse/DEFI-2933)
(sweeper fee funding). Targets `master`; dfinity#11097 follows.

## Why

The funding task shipped in dfinity#11086, but nothing makes its accounting
visible. An operator cannot see whether ckETH is still fully backed, nor
whether funding has stopped happening — and a funding that gets stuck
blocks every later one, so sweeping starves without anything else
looking wrong.

## The metrics

Eight, all under `cketh_minter_sweeper_funding_`:

| metric | |
|---|---|
| `cketh_burned_total`, `eth_spent_total` | the two sides of the
invariant: burned must never fall below spent |
| `finalized_total{status="success"\|"failure"}` | fundings that
finalized, by outcome, so both throughput and a failure rate are
expressible. Failures are expected to stay at zero — funding is a bare
transfer to an address derived from the minter's own key, with no code
to revert in — so any non-zero value wants a look |
| `burned_not_yet_spent` | how far burn runs ahead of spend |
| `gas_balance` | the prepaid-gas bound the funding decision reads |
| `low_water_mark`, `target` | the thresholds it compares that bound
against |
| `in_flight_age_seconds` | how long the outstanding funding has been
outstanding, 0 if none |

**The alert to write is `gas_balance < low_water_mark`, held for a
couple of hours.** That is "the sweeper needs gas and is not getting
it", and it catches every cause at once: a refusal for want of
deposit-backed ETH, a burn failing on an empty fee account, a funding
wedged in the pipeline, or the task not running at all. It stays quiet
on an idle minter whose sweeper is still full, which is why it is this
rather than an age-since-last-funding. Exporting the two thresholds is
what keeps it from hardcoding a number that moves with
`cketh_minimum_withdrawal_amount`.

`in_flight_age_seconds` is then the diagnostic rather than the alert:
when the predicate above fires, a climbing age says the cause is a
wedged funding, and the dashboard row says which one.

`burned_not_yet_spent` is worth one caution: for a funding that
succeeded it is the same wei `cketh_minter_total_unspent_tx_fees`
already counts, so summing the two double counts. Proposed alert
conditions live on
[DEFI-2965](https://dfinity.atlassian.net/browse/DEFI-2965).

## The dashboard

A sweeper-funding section with seven rows: the gas-paying address
derived from the minter's key, the prepaid-gas bound, the low-water mark
and target, ckETH burned and ETH spent, the funding in flight if there
is one, and the unspent burn.

It renders an in-flight funding's acceptance time as a timestamp rather
than an age, because `DashboardTemplate::from_state` must stay callable
outside a canister — `ic_cdk::api::time()` traps in unit tests — and
ages belong in metrics anyway.

A funding also travels the ordinary withdrawal pipeline, so it appears
in the withdrawal table; it is marked there rather than reading as an
unexplained ckETH withdrawal to an address nobody recognises.


[DEFI-2933]:
https://dfinity.atlassian.net/browse/DEFI-2933?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[DEFI-2965]:
https://dfinity.atlassian.net/browse/DEFI-2965?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: IDX GitHub Automation <infra+github-automation@dfinity.org>
The bytecode is a REVERT, but the transfer never reaches it: all 21'000 gas
of a bare transfer is intrinsic, so the first opcode is already out of gas.
Any non-empty code would do.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The finalization helper can miss an already-completed funding and wait indefinitely.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

rs/ethereum/cketh/minter/tests/deposit_from_cex.rs:412

  • “Any non-empty code” is too broad: a zero-cost first opcode such as STOP can succeed with zero execution gas. The selected bytecode is valid because its first PUSH1 has positive gas cost, so state that narrower condition.
    // Any non-empty code does it: all 21'000 gas of a bare transfer is intrinsic, so the first
    // opcode is already out of gas. PUSH1 0, PUSH1 0, REVERT is what is placed here, so that the
    // call would fail on its own terms too if it were ever reached with gas to spare.
    setup.set_code(&sweeper, &[0x60, 0x00, 0x60, 0x00, 0xfd]);

rs/ethereum/cketh/test_utils/src/live.rs:843

  • Setting arbitrary non-empty code does not by itself guarantee failure: for example, runtime bytecode beginning with the zero-cost STOP opcode can complete successfully even with no execution gas. Document the positive-cost opcode requirement so callers do not rely on a guarantee this wrapper cannot provide.
    /// Places code at `address`, so a plain value transfer to it no longer succeeds: with the
    /// 21'000 gas of a bare transfer there is nothing left to execute it.
    pub fn set_code(&self, address: &Address, code: &[u8]) {
        self.anvil.set_code(address, code);
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread rs/ethereum/cketh/test_utils/src/live.rs
Code beginning with the zero-cost STOP runs to completion on the nothing a
bare transfer leaves the callee, and the transfer succeeds. What fails the
transaction is a first opcode that costs something.
…d counter

The withdrawal timer finalizes in the same pass that sends, so a baseline
read once the transfer is on chain could already carry the finalization
being waited for. The event is a state the minter is either in or not.
Waiting for any FinalizedTransaction event would also be satisfied by an
ordinary withdrawal finalizing. The burn index a funding is keyed by all the
way through the withdrawal pipeline is what its receipt carries back.
Comment thread rs/ethereum/cketh/minter/tests/deposit_from_cex.rs
/// 21'000 gas of a bare transfer is intrinsic, leaving the callee none. Which makes this a
/// guarantee only for `code` whose first opcode costs something — code beginning with the
/// zero-cost `STOP` runs to completion on no gas at all and the transfer still succeeds.
pub fn set_code(&self, address: &Address, code: &[u8]) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replying to the suppressed comment on this doc comment; no node to thread onto.

Agreed. The wrapper cannot promise that placing code makes a transfer fail, only that it does for code whose first opcode costs something. Fixed in 334a51e: the doc states that condition and names STOP as the case that slips through, so nobody reaching for this reads it as a guarantee.

Comment thread rs/ethereum/cketh/test_utils/src/live.rs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The reviewed changes have no unresolved approval-blocking issues.

Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@mbjorkqvist
mbjorkqvist marked this pull request as ready for review September 4, 2026 09:26
@mbjorkqvist
mbjorkqvist requested a review from a team as a code owner September 4, 2026 09:26
@github-actions github-actions Bot added the @defi label Sep 4, 2026
@zeropath-ai

zeropath-ai Bot commented Sep 4, 2026

Copy link
Copy Markdown

No security or compliance issues detected. Reviewed everything up to 9475afb.

Security Overview
Detected Code Changes
Change Type Relevant files
Enhancement ► rs/ethereum/cketh/minter/BUILD.bazel
    Remove tests/sweeper_funding.rs from test suite
► rs/ethereum/cketh/minter/tests/deposit_from_cex.rs
    Add extensive new tests and commentary around sweeper funding and withdrawal timing considerations
Enhancement ► rs/ethereum/cketh/test_utils/src/live.rs
    Introduce funding-related constants, metrics, and helper methods for live tests (e.g., SWEEPER_ETH_SPENT, SWEEPER_BURNED_NOT_YET_SPENT, SWEEPER_GAS_BALANCE, AWAIT_DEADLINE made public)
► rs/ethereum/cketh/test_utils/src/live.rs
    Add DEPOSIT_AMOUNT constant and additional live test utilities (metric querying, minter log awaiting, ticking advancement, code deployment, ETH balance assertions)
► rs/ethereum/cketh/test_utils/src/live.rs
    Extend LiveSetup with new funding-related flows (new_funding, new_funding_with_empty_fee_account) and test-driven helpers (await_successful_funding, set_code, code, assert_no_eth_received, advance_ticks, await_minter_log, etc.)
► rs/ethereum/cketh/test_utils/src/live.rs
    Update await_funding_finalized to await_funding_finalized with max_ticks parameter and related logic
Refactor ► rs/ethereum/cketh/test_utils/src/live.rs
    Make AWAIT_DEADLINE public (pub const) and adjust related pathways accordingly
► rs/ethereum/cketh/test_utils/src/live.rs
    Refactor some function signatures (e.g., poll_until becomes pub fn poll_until) and introduce new helper methods for better test orchestration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants