feat(health): expose settlement lag so a paying-nobody chain cannot look healthy

[Scottcjn]

The outage

RustChain never settled epochs 91-174. That is 2026-03-03 to 2026-05-27, 84 consecutive days. epoch_state has no row for any of them, not settled = 0, absent.

last on-time settle	epoch 90, 2026-03-03 20:30 UTC
resumed	epoch 175, 2026-05-27 20:26 UTC
miners enrolled during the gap	183 (21-27 per epoch, 1,733 rows)
paid	nobody
RTC never emitted	126 (84 x the 1.5 fixed pot)

Settlement resumed forward from the present and never backfilled. Of the 183 miners, 31 ever attested again and 13 are active today: 83% never came back. (Correlation, not proven causation - no operator has said why they left.)

Why it ran 84 days unnoticed

Nothing was watching settlement. Everything that was being watched looked perfect:

• the epoch number is derived from wall-clock, so it kept counting up
• blocks kept being produced, so tip_age_slots stayed 0
• /health therefore reported ok: true for the entire 84 days
• node_health_monitor read only epoch and miners, both healthy

And finalize_epoch returned early on an empty miner set without writing a row or logging anything, so the outage left no trace to find afterwards.

This is the same class as the /status bug fixed in #8220: a green light that does not depend on the thing that matters. There, a 200 was not evidence of a node. Here, an advancing epoch is not evidence that anyone is being paid.

The fix

Node. _settlement_lag_epochs() reports epochs since the last settled epoch, on /epoch (reusing its existing connection) and /health.

The query is bounded by the current epoch on purpose. epoch_state still carries rows from the pre-2025-12 numbering (a stray 424 settled in Dec 2025, plus 20000-series values). An unbounded MAX() returns one of those, computes a negative lag, and reports flawless health during exactly the stall it exists to detect. I nearly shipped that version; it has its own test.

_record_unsettled_epoch() makes both no-payout paths write settled = 0 and log the reason. Safe: the authoritative replay guard inserts the same row then atomically claims 0 -> 1, so an existing unsettled row cannot block a real settlement, and nothing was credited here to be credited twice.

Monitor. NodeStatus.settlement_lag_epochs, NetworkHealth.settlement_stalled, threshold 3 (steady state is 1, so this catches a stall on day two rather than day eighty-four).

Two deliberate non-changes

• A stalled node stays online. It is answering, and miners must keep enrolling so the stalled epochs stay reconstructible from epoch_enroll.
• /health does not flip ok on lag. A 503 pulls the node from load-balancer rotation and stops miners enrolling, which is the opposite of what a settlement stall needs.

Testing

19 tests. The trickiest case is pinned against the real helper rather than a copy of its SQL, since a test that reimplements the logic guards nothing.

Both halves mutation-tested:

mutation	result
unbind the query	AssertionError: 20424 != 90
remove the monitor alert	3 failures

Reverting both restores green. Existing health suites unaffected: test_health_monitor 21, test_node_liveness_requires_node_state 14, test_rustchain_health* 13, all pass.

Not in this PR

The 84 stalled epochs are recoverable - epoch_enroll survived intact, 84/84 epochs, 1,733 rows, every one with positive weight. Distributing them is a monetary act (126 RTC to 186 wallets), not a bug fix, so it is deliberately left as a separate decision.

[github-actions]

⚠️ BCOS v2 Scan Results

Metric	Value
Trust Score	49/100
Certificate ID	BCOS-df46b432
Tier	L1 (not met)

What does this mean?

The BCOS (Beacon Certified Open Source) engine scans for:

• SPDX license header compliance
• Known CVE vulnerabilities (OSV database)
• Static analysis findings (Semgrep)
• SBOM completeness
• Dependency freshness
• Test infrastructure evidence
• Review attestation tier

Full report | What is BCOS?

---

BCOS v2 Engine - Free & Open Source (MIT) - Elyan Labs