Consensus node: startup check for next-epoch beacon key, with epoch-scoped acknowledgment flag

[zhangchiqing]

Motivation

A consensus node that loses its Random Beacon (DKG) private key cannot contribute beacon shares to consensus. Under the current (V2) signature scheme, such a node automatically falls back to staking-only signatures (CombinedSigner / CombinedSignerV3, ErrNoBeaconKeyForEpoch branch). Its votes and proposals remain valid and are included in QCs — the node keeps "participating," but contributes no beacon share. If too many nodes are in this state (fewer than RandomBeaconThreshold(n)+1 beacon-capable nodes), QC production fails and finalization halts even with full staking weight online.

Today, the --require-beacon-key startup check only covers the current epoch. The problems:

Problem 1: missing next-epoch key is discovered at the epoch boundary, not at startup

--require-beacon-key=false downgrades the current-epoch check to a warning, and the next epoch's key is not checked at all. A node restarted during the committed phase without a valid next-epoch key (e.g. after a dynamic bootstrap that didn't restore /var/flow/data/secrets) starts up happily. The operator finds out only when the epoch transitions and the node silently stops contributing beacon shares.

Problem 2: the leniency flag never expires

--require-beacon-key=false is a standing configuration. An operator who set it while recovering from a past incident typically leaves it in the config forever. Every subsequent missing-key event then passes as a warning that nobody reads. The escape hatch outlives the incident it was created for.

Problem 3: rolling restarts in the non-staking phase can silently kill finalization fleet-wide

Observed incident: restarting consensus nodes one-by-one during the committed phase, with the secrets DB not preserved, caused each restarted node to lose its beacon key for the next epoch. Each node started fine (Problem 1), so the rollout proceeded. Once too many nodes were affected, beacon reconstruction failed and finalization halted.

Key mechanics that make this insidious:

• A plain restart during the committed phase is safe — the DKG private share is persisted when the DKG completes (setup phase). The danger is restart with secrets loss, or restart during the setup phase late enough that the node irrecoverably misses its DKG window (DKGStateFailure).
• After the epoch is committed, a missing next-epoch key is unrecoverable: the DKG is over, and the only recovery path (EFM key recovery, copying the current-epoch key forward) runs automatically at startup — so if it hasn't restored the key, no key is coming.
• Since beacon-less nodes still produce valid staking-only votes, the failure is invisible until the threshold is crossed.

The operator needs the node itself to refuse to start in this state, so that a rolling rollout halts at the first affected node instead of marching through the committee.

Problem 4: crash-based enforcement needs a self-expiring escape hatch

If the node crashes on startup in this state, the operator needs a way to bring it back immediately: it still holds a valid current-epoch staking key (it can keep voting this epoch and use it to recover DKG key for next epoch), and it must be running to participate in the next DKG and heal for the following epoch. (Without any bypass, the node self-heals only at the epoch boundary — the crash loop lasts the remainder of the committed phase and its current-epoch voting weight is wasted.)

A boolean bypass flag (--skip-...=true) reintroduces Problem 2: forgotten, it permanently disables the protection.

Problem 5 (context): external monitoring cannot substitute

Under V2, QCs contain no per-signer beacon attribution (RandomBeaconSigners is never populated), and the staking-only fallback is valid by design — so a keyless node is indistinguishable from a healthy one in finalized chain data until one watches proposer signature types over time. Chain-level monitoring (planned separately, via access-node metrics on finalized blocks) detects key loss within minutes after it manifests, but only startup enforcement can catch a doomed next-epoch key before the boundary.

Proposed solution

Rework the startup beacon-key verification (module/dkg/verification.go, invoked from cmd/consensus/main.go) into three checks. Each check has a subject epoch E:

#	Check	Subject epoch	Default result on failure
1	Current-epoch key exists, is marked safe, and matches the node's key share in the protocol state	current	crash
2	Next-epoch key exists, is safe, and matches EpochCommit.DKGParticipantKeys — evaluated only when the next epoch is committed and this node is a DKG participant	next	crash
3	Next-epoch DKG state is DKGStateFailure while still in the setup phase (the node has already irrecoverably missed its DKG window)	next	crash

Explicit pass conditions (no flag needed): next epoch not yet committed (the DKG may still deliver a key); node not in the next epoch's DKG committee; key already restored by BeaconKeyRecovery (which runs in an earlier startup module, so a crash here means genuinely unrecoverable).

New flag: --acknowledge-missing-beacon-key-up-to-epoch=N (uint64, default 0 = nothing acknowledged)

• Any check whose subject epoch ≤ N downgrades from crash to an error-level log (potential-configuration-problem=true) and the node starts degraded.
• Self-expiring: epoch counters are monotonic and checks only ever concern the current or next epoch, so a stale N can never shield a later epoch. An operator who forgets to remove it is automatically re-protected — the exemption expires, never the protection.
• One value covers combined incidents (e.g. both keys wiped during the committed phase: current and next subject epochs both ≤ N).

Deprecate --require-beacon-key

Still accepted for backward compatibility: false keeps its legacy meaning for the current-epoch check only (warn + start) and logs a deprecation warning pointing to the new flag. It has no effect on the next-epoch checks — the standing posture can no longer weaken them. (Note this is a deliberate behavior change for existing =false deployments: next-epoch failures become fatal.)

Crash messages are the notification and the runbook

Each failure produces a distinct fatal log that (a) names which epoch's key failed and why it is unrecoverable, (b) prints the exact recovery command with the value filled in, e.g. --acknowledge-missing-beacon-key-up-to-epoch=10, and (c) for next-epoch failures, warns to halt any in-progress rolling restart of consensus nodes.

How this addresses each problem

Problem	Addressed by
P1: next-epoch key loss found at the boundary	Check 2 crashes at startup during the committed phase — before the boundary
P2: leniency flag never expires	New flag is epoch-scoped and self-expiring; the bool is deprecated and no longer affects next-epoch checks
P3: rolling restart silently halting finalization	First doomed node crash-loops with a fatal message; the rollout stops at node 1 instead of reaching the beacon threshold
P4: need an escape hatch without a permanent one	The acknowledgment flag restores the node immediately (it can vote in the current epoch and run the next DKG), and auto-expires
P5: external monitoring can't substitute	Complementary: startup enforcement covers pre-boundary detection; the planned access-node monitor covers post-manifestation detection fleet-wide

Scenario matrix (for verifying coverage)

Restart at various times, with various key states. Default flags unless noted.

Restart timing	Key state	Outcome
any phase	all keys intact	starts normally (plain committed-phase restart is safe: the key was persisted at DKG completion)
staking/setup phase	current-epoch key missing	crash; --require-beacon-key=false (deprecated) or acknowledgment flag → degraded start; heals via next DKG
setup phase	next-epoch DKG already DKGStateFailure	crash (check 3) — the chance to derive the key is gone; acknowledgment flag → degraded start
setup phase, DKG ongoing	no next-epoch key yet	starts normally — the DKG may still deliver
committed phase	next-epoch key missing/invalid, node in next DKG committee	crash (check 2), regardless of any standing flag; acknowledgment flag with N ≥ next epoch → degraded start, message warns to halt rolling restarts
committed phase	next-epoch key missing, recoverable via EFM recovery	starts normally — BeaconKeyRecovery restored it before the check ran
committed phase	node not in next epoch's DKG committee	starts normally — nothing to verify
after epoch boundary (new staking phase)	previous next-epoch key was lost	starts degraded via existing leniency; runs the DKG for the following epoch and heals

Accepted tradeoff

An operator intentionally restarting a single node during the non-staking phase, with lost secrets, will now hit the crash and must add --acknowledge-missing-beacon-key-up-to-epoch=<epoch> to proceed. This is deliberate friction: the crash message states exactly what to do, and the cost (one extra restart with a flag) is small compared to the failure mode it prevents (fleet-wide finality halt). Operators with intact secrets are unaffected — plain restarts never trip these checks.

Out of scope / follow-ups

• Network-wide monitoring (separate work, access-node side): per-node proposer signature-type metrics over finalized blocks, giving a live view of beacon-capable committee weight vs. the reconstruction threshold (RandomBeaconThreshold(n)+1). This detects mid-epoch key loss fleet-wide within minutes of manifestation; the startup checks here cover pre-boundary detection.
• Per-node runtime signing metrics on consensus nodes: dropped in favor of the access-node monitor (deploying to one AN vs. the whole consensus fleet).

---

Produced in collaboration with an AI assistant (Kimi).