feat: Add a monotonic consumed_cycles_as_counter to CanisterMetrics

[mraszyk]

CanisterMetrics::consumed_cycles behaves like a gauge: a prepayment raises it and
the matching refund lowers it again. That makes it awkward to build monitoring on
top of, which is why consumed_cycles_by_use_cases already has a monotonic
consumed_cycles_by_use_cases_as_counters twin, only bumped once the refund of a
prepayment is known (#9922).

This adds the same twin for the scalar total, plus the migration that the
by-use-case counters never got -- #9922 deferred it, noting that "a proper cutoff
point needs to be introduced to handle outstanding callbacks that might have been
created before the metric introduction" -- so the new counter starts out with the
full history rather than from zero.

No cutoff point is needed

The two metrics are tied by an invariant:

consumed_cycles - outstanding prepayments == consumed_cycles_as_counter

Only Instructions and RequestAndResponseTransmission charges are ever refunded
(they are the only two CyclesUseCaseRefundableKinds), and an outstanding
prepayment of either is always recorded in the replicated state:

• in the Callback of a call that has not been responded to
  (prepayment_for_response_execution and prepayment_for_call_transmission --
  exactly the two amounts charged by SandboxSafeSystemState::push_output_request);
  or
• in the prepaid_execution_cycles of an aborted execution or an aborted
  install_code. (A paused execution records no prepayment of its own: a paused
  response execution is paid for by the callback the task carries, and any other
  paused execution holds its prepayment in memory only -- see below.)

SystemState::outstanding_prepayments() sums them up, so the counter can be derived
from the gauge exactly, with no residual over-count. And since the invariant
holds at all times -- not just before the counter is first observed -- deriving it
is idempotent: the backfill can be redone in every round and is self-healing if a
downgrade drops the counter.

Two details worth calling out:

• prepayment_for_call_transmission is zero for callbacks created before April
  2026, and apply_initial_refunds falls back to
  prepayment_for_response_transmission for those; outstanding_prepayments
  mirrors that fallback, or the affected canisters would be off.
• A paused or aborted response execution prepays nothing of its own (its
  prepaid_execution_cycles is zero); it is paid for by the callback that the task
  carries, which was unregistered from the CallContextManager when the response
  was popped. So the two sources stay disjoint.

One behaviour change: a subnet split now refunds the dropped install_code

There is exactly one place where a recorded prepayment used to leave the state
without a refund, and it had to be closed for the invariant to hold.
drop_in_progress_management_calls_after_split() drops the canister's
AbortedInstallCode task on subnet B, and with it the prepaid_execution_cycles
recorded in the task -- previously with no accounting at all, silently leaving the
canister charged for it.

remove_aborted_install_code_task() now returns the prepayment and the caller
refunds it in full. That is what the canister is owed: an aborted execution discards
the slices it had already run and starts over when retried, and this one is never
retried, while subnet A' rejects the corresponding call and unwinds the operation
there as well. The refund is deterministic across subnet B's replicas and conserves
cycles -- they move back from consumed to the canister's balance -- and the split
path returns early from execute_round, before any of the cycle-conservation
assertions.

It also keeps the metrics right: the refund lowers the gauges by the prepayment and
adds nothing to the monotonic counters, which is exactly what should happen when
nothing was consumed.

This is the one part of the PR that changes canister balances, so it deserves a
closer look than the rest.

Where the backfill runs

On checkpoint rounds only, after abort_all_paused_executions. A paused execution
is ephemeral, so its prepayment is not part of the replicated state (except for a
paused response execution, see above); aborting materializes it into the canister's
task queue. Canisters that still have a paused execution are skipped and picked up
at the next checkpoint round.

The sweep only takes a mutable reference to a canister whose counter actually
changes, so it degenerates to a read-only pass once every canister has been
backfilled. It is unconditional and idempotent, like
SubnetMetrics::migrate_outcalls_cycles_to_use_cases in the same function.

Scope, and the second behaviour change: what the existing gauge now exports

Like the gauge it mirrors, the counter covers everything except HTTPS outcalls,
which are only tracked at the subnet level (and, for the canister, in
consumed_cycles_by_use_cases_as_counters). This keeps it substitutable for the
gauge in the subnet-wide aggregate.

No new Prometheus metric is added. The existing
replicated_state_consumed_cycles_since_replica_started gauge (and its
replicated_state_consumed_cycles alias) now exports the monotonic subnet-wide
total -- the subnet-level aggregate, which is monotonic already (subnet-level use
cases are only ever charged, never refunded, and a deleted canister's consumption
moves into consumed_cycles_by_deleted_canisters), plus the canisters'
consumed_cycles_as_counter -- in place of the certified
SubnetMetrics::consumed_cycles_total_including_canisters it used to carry. The two
differ by exactly the outstanding prepayments, i.e. the gauge tracks the certified
total closely and is simply better behaved.

Reusing the name is deliberate, per review:

• It stays a gauge, not a Prometheus Counter. The value is recomputed from the
  state on every observation, so a replica that restarts mid-interval and resumes
  from the last checkpoint re-exports a lower value than it had reached before; a
  Counter's reset() + inc_by() would report that as an increase by the whole
  checkpoint value. An online subnet split lowers it for the same reason (the
  migrating canisters' consumption goes to the other subnet, and retaining it here
  would count it on both). So it still needs the high water mark treatment that the
  non-monotonic value did -- it is just a much better behaved gauge in the average
  case.
• Since it must be a gauge anyway, exporting the monotonic value under the old name
  saves rewriting every rule and dashboard (including the public dashboard) that
  already scrapes it.

The canister deletion path needs no change: a canister can only be deleted while
Stopped and with empty queues, so it has no outstanding prepayment and its gauge
equals its counter. Documented at the call site.

CanisterMetrics::consumed_cycles and
SubnetMetrics::consumed_cycles_total_including_canisters themselves are untouched,
so the certified /subnet/<subnet_id>/metrics total is unaffected.

Notes for the reviewer

• The first checkpoint round after the upgrade Arc::make_muts every canister to
  backfill it. That is a one-time cost on a round that already rewrites every
  canister.pbuf (the new proto field forces it). Happy to spread it over several
  checkpoint rounds if preferred.
• Renaming _as_counter to _monotonic throughout (CanisterMetrics, the
  persisted protobuf field, the locals) is not in this PR; it is a mechanical
  but wide rename of a persisted field name, better done on its own. Say the word
  and I will fold it in here instead.
• The same backfill trick would also work for
  consumed_cycles_by_use_cases_as_counters -- split outstanding into its
  Instructions and RequestAndResponseTransmission parts and it falls out. Left
  for a follow-up.
• Locally I ran the directly affected targets (replicated_state, state_layout,
  state_manager, execution_environment, canonical_state, embedders,
  cycles_account_manager, messaging, replay, state_machine_tests,
  protobuf, types) plus rust-lint.sh. The NNS/SNS/registry integration tests,
  which reach this change only through the regenerated protobuf bindings, exhaust
  the disk on my machine, and the two golden-state tests need internal SSH access;
  those are left to CI.

Tests

• The counter update rules, per use case and cost schedule.
• All three sources of outstanding prepayments, including the legacy
  prepayment_for_call_transmission fallback and the response case that must not be
  double counted.
• Exactness and idempotence of the backfill, and that it skips a paused execution.
• Protobuf round-trip, with and without the new field (a checkpoint predating it).
• Scheduler tests: the invariant after a round with a call in flight, the backfill on
  a checkpoint round (and that an ordinary round does not do it), the same for a
  canister with a paused DTS execution, and that the exported gauge equals the
  certified total net of the outstanding prepayments
  (consumed_cycles_total_is_exported_net_of_outstanding_prepayments).
• The subnet split refund, both in isolation
  (refunds_prepayment_of_aborted_canister_install_dropped_after_split) and in
  online_split, whose expected post-split state now accounts for it.

[Copilot]

🟡 Changes recommended

Online subnet splits can decrease the exported counter and invalidate the prepayment invariant.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a persistent monotonic consumed-cycles counter, migration logic, Prometheus export, and coverage across state loading and scheduling.

Changes:

• Tracks and serializes consumed_cycles_as_counter.
• Backfills historical values from gauges and outstanding prepayments.
• Exports and tests the aggregate counter.

File summaries

File	Description
rs/state_manager/src/tip.rs	Serializes the counter.
rs/state_manager/src/checkpoint.rs	Restores the counter.
rs/state_layout/src/state_layout/tests.rs	Tests protobuf compatibility.
rs/state_layout/src/state_layout/proto.rs	Converts the protobuf field.
rs/state_layout/src/state_layout.rs	Adds persisted state.
rs/replicated_state/src/metrics.rs	Exports the aggregate counter.
rs/replicated_state/src/canister_state/tests.rs	Tests accounting and migration.
rs/replicated_state/src/canister_state/system_state.rs	Implements counter accounting and derivation.
rs/protobuf/src/gen/state/state.canister_state_bits.v1.rs	Updates generated protobuf types.
rs/protobuf/def/state/canister_state_bits/v1/canister_state_bits.proto	Defines the protobuf field.
rs/execution_environment/src/scheduler/tests/metrics.rs	Tests scheduler migration and export.
rs/execution_environment/src/scheduler.rs	Runs checkpoint backfills.
rs/execution_environment/src/canister_manager.rs	Documents deletion accounting.

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mraszyk]

Good catch, fixed.

drop_in_progress_management_calls_after_split() now settles the prepayment of the dropped AbortedInstallCode instead of discarding it: remove_aborted_install_code_task() returns the prepaid_execution_cycles and the caller observes them as a refund of zero. That leaves the balance and both consumed-cycles gauges untouched (nothing is actually returned to the canister) and moves the prepayment onto the monotonic counters, which is where a prepayment belongs once its refund is known. The invariant is restored in the same round, and the Instructions by-use-case counter gets the amount too -- the checkpoint sweep alone would only have caught up the scalar.

One correction to the ordering in the report: a splitting batch returns early from StateMachineImpl::execute_round (rs/messaging/src/state_machine.rs), so the split round runs no execution and the backfill sweep never runs in it. The observable effect was as described though -- the split checkpoint would have carried a counter lagging by the dropped prepayment until a later checkpoint round bumped it. Note also that the lag was in the safe direction (the counter only ever caught up), so this was an exactness bug rather than a monotonicity one.

Not refunding the cycles to the canister balance is deliberate: the execution was paid for and is being discarded, and changing balances during a split is a consensus-critical semantic change that does not belong in this PR.

Covered by settles_prepayment_of_aborted_canister_install_dropped_after_split.

[mraszyk]

Correction to my reply above: the prepayment is now refunded in full rather than settled as consumed (44be0cb).

The canister has nothing to show for those cycles -- an aborted execution discards the slices it had already run and starts over when retried, and this one is never retried, while subnet A' rejects the corresponding call and unwinds the operation there too. So charging for it was wrong, and my "not refunding is deliberate" argument does not hold up: the refund is deterministic across subnet B's replicas and conserves cycles (they move back from consumed to the balance), and the split path returns early from execute_round before any of the cycle-conservation assertions.

Both variants settle the prepayment and so restore the invariant equally -- G' - o' = (G-p) - (o-p) = G - o = C -- but the refund lowers the gauges by the prepayment and adds nothing to the monotonic counters, which is what should happen when nothing was consumed.

Test renamed accordingly, and online_split in rs/replicated_state/tests/replicated_state.rs now expects the refund in its post-split state.

[mraszyk]

The premise is right -- an online split does lower this total -- but neither suggested remedy is the one we want, so this stays as is, now with a comment explaining why.

Transferring the migrated canisters' consumption into a monotonic subnet accumulator would count it twice: subnet B keeps those canisters and their counters, so subnet A' would report consumption that is still being reported next door. It would also change SubnetMetrics::consumed_cycles_total(), which feeds the certified /subnet/<subnet_id>/metrics from CertificationVersion::V29 -- not something to change as a side effect of adding a metric.

A process-local baseline would keep the exported number monotone at the cost of permanently diverging from the state it reports, and would silently mask any genuine regression in the underlying value. The debug_assert in migrate_consumed_cycles_to_counters exists precisely to catch those.

What is left is a counter reset at an online split, which Prometheus handles: rate()/increase() recognise the reset and lose one interval's increment. Worth accepting, because the drop is not an artefact -- the certified consumed_cycles_total_including_canisters that this metric mirrors drops for exactly the same reason, as does the existing replicated_state_consumed_cycles_from_replica_start_as_counters, which folds the same per-canister counters into a CounterVec. Making this one metric split-proof in isolation would leave it inconsistent with both.

[Copilot]

🟡 Changes recommended

The migration invariant documentation contains two materially inaccurate descriptions of paused and split execution handling.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 15/15 changed files
• Comments generated: 2
• Review effort level: Balanced

[mraszyk]

Correct, fixed. The bullet now reads "in the prepaid_execution_cycles of an aborted execution or an aborted install_code" -- PausedExecution has no such field.

Added a parenthetical right after the list so the paused cases are accounted for where the reader meets them, rather than only in the Returns None paragraph further down: a paused response execution is paid for by the callback the task carries (covered by the first bullet), and any other paused execution holds its prepayment in memory only, which is why the method returns None for it.

[mraszyk]

Right, that paragraph was left over from the settle-as-consumed version. Reworded:

The only way such a prepayment leaves the state other than through the refund that its execution or response issues is an aborted install_code dropped by a subnet split; that path refunds it in full, see Self::drop_in_progress_management_calls_after_split.

So it is no longer framed as an exception to the invariant -- it is just an unusual place where the refund is issued -- while still pointing a reader who wonders what happens when the task disappears at the code that handles it.

[Copilot]

🔵 Needs a closer look

The consensus-critical cycle accounting and checkpoint migration require final human validation despite comprehensive tests.

Review details

• Files reviewed: 15/15 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[alin-at-dfinity]

Logic and tests look sane.

But man, is this verbose! I'm fine with leaving any temporary migration, assert and test code (and their comments) as is, but I would try to cut down on noise for everything that's here to stay.

[alin-at-dfinity]

Sorry Claude, this is stupidly verbose. migrated_consumed_cycles_as_counter can be trivially inlined; and it's not used anywhere else.

[mraszyk]

Done, inlined in 2339379. migrate_consumed_cycles_to_counter now reads:

pub fn migrate_consumed_cycles_to_counter(&mut self) -> bool {
    let Some(outstanding) = self.outstanding_prepayments() else {
        return false;
    };
    // `max` rather than a plain assignment: the counter must never go down, not
    // even if a saturating subtraction somewhere made the gauge lag behind it.
    self.canister_metrics.consumed_cycles_as_counter = self
        .canister_metrics
        .consumed_cycles_as_counter
        .max(self.canister_metrics.consumed_cycles - outstanding);
    true
}

The doc comment of the removed method moved onto it, with the Returns None paragraph turned into the Returns false one.

[alin-at-dfinity]

I believe that everything after the first comment line can be safely dropped.

Suggested change

	// Refund the execution cycles prepaid for the dropped `install_code` in full.
	// The canister has nothing to show for them: an aborted execution discards the
	// slices it had already run and starts over when retried, and this one is
	// never retried -- subnet A' rejects the corresponding call, unwinding the
	// operation there as well.
	//
	// This also keeps the consumed cycles metrics consistent. Dropping the task
	// without refunding would leave the prepayment in the gauges with nothing left
	// in the state to account for it, breaking the invariant on
	// `Self::outstanding_prepayments`; and the monotonic counters, which only
	// account for a prepayment once its refund is known, would never see it at
	// all. A full refund lowers the gauges by the prepayment and adds nothing to
	// the counters, which is exactly right: nothing was consumed.
	if let Some(prepaid_execution_cycles) = self.task_queue.remove_aborted_install_code_task() {
	self.refund_cycles(prepaid_execution_cycles, prepaid_execution_cycles);
	}
	// Refund the execution cycles prepaid for the dropped `install_code` in full.
	if let Some(prepaid_execution_cycles) = self.task_queue.remove_aborted_install_code_task() {
	self.refund_cycles(prepaid_execution_cycles, prepaid_execution_cycles);
	}

Better yet, the one sentence can be added to the first paragraph of the existing comment, something like "Remove aborted install code task and fully refund the [prepaid] cycles."

[mraszyk]

Done in 2339379, taking the "better yet" variant -- the sentence is now part of the existing first paragraph:

// Remove aborted install code task and fully refund the prepaid execution
// cycles.
//
// Note that this cannot be a paused install code task, because we abort all
// paused tasks before triggering the split.
if let Some(prepaid_execution_cycles) = self.task_queue.remove_aborted_install_code_task() {
    self.refund_cycles(prepaid_execution_cycles, prepaid_execution_cycles);
}

[alin-at-dfinity]

"As counter" is not a great name. It does mirror "by use cases as counters", but while the CanisterMetrics field does kind of act like a Prometheus counter, it is it is just a field in a struct (nothing to do with Prometheus counters).

And the actual Prometheus metric should emphatically not be a counter (as hinted to by the fact that we reset it to zero and then increment it every time): if the replica process restarts half-way through a checkpoint interval and then resumes from the checkpoint, the scraped time series will increase past the CanisterMetrics "counter" at the checkpoint height (say N), then reset back to N and continue from there. From the POV of Prometheus, the underlying value just jumped from N+k to 0 and then back to N, an increase of N cycles.

I.e. definitely make this a gauge. And ideally drop the _as_counter from everywhere. Just name it _monotonic.

Even better, there is already code that aliases the replicated_state_consumed_cycles_since_replica_started / replicated_state_consumed_cycles_from_replica_start to names that drop their misleading suffixes. Those alias names are not used anywhere (dashboards, alerts or other rules). So we could grab the chance to drop the unused aliases and export the monotonic counters cleanly as replicated_state_consumed_cycles[_by_use_case].

[alin-at-dfinity]

And actually, since the new metric must be a gauge, we may as well avoid adding a new metric and instead simply export the new monotonic value under the old name(s).

That way we save significant effort on rules and dashboards (including the public dashboard): as said, the value is potentially not actually monotonic across replica restarts, so computing rates of increase and whatnot still requires the same high water mark approach we have for the current gauge. It's merely a much better behaved (in the average case) gauge than the existing one.

[mraszyk]

Both points taken, and the second one supersedes the first, so 2339379 does exactly that: the new Prometheus metric is gone, and the monotonic total is now exported under the existing replicated_state_consumed_cycles_since_replica_started gauge (and, via the existing register_alias, replicated_state_consumed_cycles), in place of the certified consumed_cycles_total_including_canisters it used to carry.

// The monotonic counterpart of the certified
// `SubnetMetrics::consumed_cycles_total_including_canisters()`, from which it
// differs by exactly the prepayments whose refund is not known yet: the
// subnet-level aggregate (monotonic already, ...) plus the canisters' monotonic
// counters. Exported in place of the certified total, which it tracks closely,
// so that existing rules and dashboards need no change.
//
// Still a gauge, not a Prometheus `Counter`: the value is recomputed from the
// state, so a replica resuming from a checkpoint re-exports a lower value than
// it had reached before; and an online subnet split hands the migrating
// canisters' consumption over to the other subnet (retaining it here would
// count it on both). Both need the same high water mark treatment as the
// non-monotonic value did.
self.consumed_cycles.set(
    (state.metadata.subnet_metrics.consumed_cycles_total()
        + consumed_cycles_by_canisters_as_counter)
        .get() as f64,
);

Your N+k → 0 → N argument is the decisive one and I had missed it: a Counter whose value is recomputed from the state on every observation is not a counter at all. The reset() + inc_by() pattern only works for a value the process itself accumulates, and this one is derived from a checkpoint the process may be resumed from. So no rules or dashboards to write, and no new metric to roll out -- just a strictly better behaved gauge under the name everything already scrapes. The help string now says so ("Monotonic, except across replica restarts and subnet splits: a prepayment is only accounted for once the refund it produces is known.").

The aliases stay as they are: since we are reusing the old names rather than introducing new ones, dropping them is unrelated to this PR (and they are still the in-flight half of the earlier rename).

What I have not done here is the _as_counter → _monotonic rename, in CanisterMetrics, the protobuf field and everywhere else. It touches the persisted field name and a good deal of surrounding code, so I would rather do it as its own commit/PR than bury it in this one -- happy to do it right away if you would prefer it in here.

Test-wise: consumed_cycles_total_as_counter_is_exported became consumed_cycles_total_is_exported_net_of_outstanding_prepayments, asserting exported == certified_total - outstanding; and consumed_cycles_gauge_accounts_for_all_subnet_level_use_cases no longer stages a canisters' part via refresh_consumed_cycles (the gauge folds the canisters directly now), with the canisters' part covered by the scheduler test above.

[Copilot]

🔵 Needs a closer look

The consensus-state migration and canister-balance change require human review, and metric contract documentation remains inconsistent.

Review details

• Files reviewed: 16/16 changed files
• Comments generated: 2
• Review effort level: Balanced

[mraszyk]

Right, the description was stale -- the new metric was dropped in 2339379 in favour of reusing the existing name, and the description still described the earlier revision. Updated: it now has a "Scope, and the second behaviour change: what the existing gauge now exports" section stating explicitly that no new Prometheus metric is added and that replicated_state_consumed_cycles_since_replica_started (and its replicated_state_consumed_cycles alias) now carries the monotonic total in place of the certified consumed_cycles_total_including_canisters, along with why it must stay a gauge rather than become a Counter.

The code is the intended state here, not the description: the reset-and-increment pattern a Counter would need is wrong for a value recomputed from the state on every observation, and reusing the old name spares every rule and dashboard that already scrapes it.

[mraszyk]

Both stale, fixed in a97772f.

SubnetMetrics::consumed_cycles_total_including_canisters:

/// This is the certified total: the state tree at `/subnet/<subnet_id>/metrics`
/// (from certification version `V29`) reads it here. The
/// `replicated_state_consumed_cycles_since_replica_started` gauge deliberately
/// exports the monotonic counterpart instead -- this same total, net of the
/// prepayments whose refund is not known yet -- see
/// `ReplicatedStateMetrics::observe`.

And the test helper, which still needs the refresh, just not for observe any more -- the assertions compare the exported gauge against the certified total:

/// Observes the state metrics at `height`, having first refreshed
/// `SubnetMetrics::consumed_cycles_total_including_canisters`, the derived
/// certified total that the assertions below compare the exported gauge against.
/// Production refreshes it on every `commit_and_certify`, which this harness never
/// does.

[Copilot]

🔵 Needs a closer look

The consensus-critical migration and subnet-split balance change require final human validation.

Review details

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Balanced