Keep a devnet across long pauses: catch up to wall clock with Yano (companion, yano-only), with step progress in the CLI

[satran004]

Keep a devnet across long pauses. Close the laptop for the weekend or stop DevKit for days, and the same chain comes back: contracts, wallets, stake and governance state are kept, and no reset is needed.

Problem

A Haskell block producer can only forge while its tip is less than one stability window (3k/f slots) behind wall clock: 5 minutes on the default devnet, 21 s with --epoch-length 40. After a longer pause the chain stalls for good, and reset was the only way out:

• Lid closed while DevKit runs. Processes freeze and resume on wake with a stale tip. The node stays up and answers queries, but never forges again. Transactions are accepted into the mempool and never confirmed.
• Stopped, started days later. start waits for a block that never comes, so Yaci Store, socat and Ogmios are not started either.
• Yano-only. Yano keeps producing, but jumps over the missed epochs in one block ("Processing epoch boundary: 3 → 6"). Store then has no protocol parameters for the skipped epochs, and reward calculation fails.

What happens now

Companion: the Haskell node restarts as a relay, Yano follows its chain, then Yano backfills sparse empty blocks to wall clock. The backfill is one block per forecast window plus one at every epoch boundary, so rewards, snapshots and governance run for each epoch that passed. The relay adopts the blocks and restarts as the block producer.
Yano-only: on start, Yano first runs without forging, backfills, then starts live.

The devnet stopped producing blocks (the machine slept?)
⏸  Devnet was idle for 1m 10s (tip slot 1,067, now slot 1,137, 1 epoch behind).
💡 Catching up to wall clock with Yano (sparse backfill, about 3 blocks) ...
  [1/4] ✅ Haskell node → relay ................. tip slot 1,067 (1s)
  [2/4] ✅ Yano following the chain ............. at slot 1,067 (4s)
  [3/4] ✅ Backfill to wall clock ............... 2 blocks to slot 1,146 (12s)
  [4/4] ✅ Haskell node back as block producer .. forging at slot 1,156 (1s)
✅ Devnet resumed in 19s at slot 1,156, epoch 11. Your contracts, wallets and stake are unchanged; 1 epoch passed.

Triggers

Trigger	When
start	automatic: catches up before Store and the other services start, when the lag is past ¾ of the window
watchdog	automatic, companion: checks the lag every 30 s while the devnet runs (the closed-lid case) and redraws the shell prompt afterwards
catch-up [--force]	on demand; a no-op on a live chain unless --force
POST /local-cluster/api/admin/devnet/catch-up?force=false	on demand, for scripts/CI
GET /local-cluster/api/admin/devnet/chain-lag	tip, wall-clock slot, lag, window, stalled (for the viewer, MCP, scripts)

devnet.auto.catch.up=false turns off the automatic part: start and the watchdog then only print a catch-up hint.

Changes

Catch-up (localcluster/catchup/)

• ChainLag: lag, forecast window, thresholds, estimates.
• DevnetCatchUpService: companion and yano-only flows. Failures always leave the devnet with Yano stopped, the original topology and the Haskell producer running.
• DevnetStallWatchdog: armed and disarmed by the ClusterStarted / Stopped / Deleted events. stop waits for a catch-up that is in progress.
• NodeControl, implemented by ClusterStartService.

Yano

• YanoService takes a run mode (LIVE, PAST_TIME_TRAVEL, FOLLOW, CATCH_UP) and passes every property as an env var too.
• CATCH_UP uses a 1-hour block time, so no live block can land at the wall-clock slot before the backfill.
• Warns when the installed Yano is not the pinned version: download keeps an existing binary, and a machine here was still on pre6.

Node restarts: cardano-node is stopped with SIGTERM to the node process instead of a forced kill (also on stop). A forced kill left the ChainDB unclean, and the next start revalidated every chunk: about 24 s, growing with the chain. A clean stop takes 0.1 s and the restart about 1.5 s.

Progress UI (util/progress/ConsoleProgress)

• Numbered steps, a spinner, a bar with a percentage when the total is known, and each step's duration.
• In a terminal the line is redrawn in place; everywhere else (Docker logs, pipes, the admin API) the steps are plain lines. YACI_PROGRESS=plain forces plain lines.
• While a step line is live, other console output is moved above it instead of being appended to it.
• Also used for the companion first-run relay sync, Yano and Yaci Store start-up, Store's sync to the tip, and downloads.

Misc

• RelaySyncWaiter can wait for a slot and report each tip read to a listener.
• getTip reports errors through the caller's writer, which stops the "Find tip error" spam from polling callers, the admin API and MCP.

Docs: "Resuming After a Long Pause" in yano-node-modes.mdx, catch-up in commands.mdx, and the property in application.properties.

Verification

Default and short-epoch companion devnets and yano-only devnets, Yaci Store 2.0.6 native:

Scenario	Result
Lid closed: all processes SIGSTOPped 6.5 min (default devnet) and 70 s (--epoch-length 100), with Store, submit-api, Ogmios and Kupo running	watchdog caught up on its next check; Ogmios, Kupo and Store reconnected and followed; SDK e2e 13/13 afterwards (Store submits through submit-api)
Stopped ~7 min, then start (default devnet)	resumed in 18 s, then Store and the other services started
Yano-only stopped across 2–4 epochs, then start	one boundary per epoch (3→4 … 6→7), protocol params present for every epoch, SDK e2e 13/13
catch-up endpoint on a live chain	no-op, "Nothing to catch up"
Progress rendering under a pseudo-terminal (script)	companion create, Ogmios download, watchdog catch-up, and foreign output during a live step
Unit tests	62 pass (ChainLag, slot-target RelaySyncWaiter, Yano run-mode config/env, installed Yano version, progress component)

Limitations

• Companion catch-up needs activeSlotsCoeff 1. Yano's catch-up on an existing chain does not check VRF eligibility; that needs a Yano change.
• Yano-only with the lid closed: Yano forges as soon as the machine wakes, before DevKit can react. That also needs a Yano change (backfill before a live block when the gap exceeds the window). Stopping the devnet before a long pause avoids it.
• Local multi-node, yano-primary and haskell-only are not covered.

Design notes: ADR-0017 (kept locally with the other ADRs).

🤖 Generated with Claude Code

[satran004]

Verdict: CHANGES REQUESTED

Review of PR #196 at 2d4dbe7506d38558da105a85390c1720be0a3e1b, using the ADR PR reviewer process against the implementation and local ADR-0017. This is an implementation PR, so implementation changes are in scope. The separate FOLLOW/IDLE/CATCH_UP modes, graceful process cleanup, and explicit unsupported-mode checks are useful safeguards. Three remaining paths bypass the intended cancellation or epoch-continuity guarantees.

F1 [P1] Propagate catch-up cancellation to the enclosing companion startup

Location: applications/cli/src/main/java/com/bloxbean/cardano/yacicli/localcluster/ClusterStartService.java:496–499, and the continuation at lines 222–237.

Problem: catchUpOnRestart() only logs an unsuccessful result. When stop or reset cancels a catch-up running inside startCluster(), the enclosing startup still calls startSubmitApi(), registers the new process, and returns RunStatus(true, ...). The catch-up clears its cancellation interrupt on exit, so nothing stops this continuation. stopCluster() only waits for the inner catch-up's inProgress flag: the continued startup can therefore race its process-list cleanup or a reset.

Evidence: A temporary probe exercises the real ClusterStartService.startCluster() with its catch-up collaborator returning Result(false, "Catch-up cancelled", ...). It prints Catch-up cancelled, then Started submit api, and returns stared=true. No production code was replaced in this caller-path probe; process creation and the catch-up collaborator were mocked.

Requested revision: Carry cancellation/fatal startup failure through to startCluster() and abort before any subsequent process creation or successful startup result. Ensure stop/reset waits for the enclosing startup operation as well as the inner catch-up. Add a concurrent start → catch-up → stop/reset regression proving no process is started or registered after shutdown completes.

F2 [P1] Keep Yano non-forging when the auto-off tip check fails

Location: applications/cli/src/main/java/com/bloxbean/cardano/yacicli/localcluster/catchup/DevnetCatchUpService.java:427–440 (startYanoOnlyWithoutCatchUp).

Problem: With devnet.auto.catch.up=false, a failed IDLE startup/readiness check or a null second tip read falls through to starting LIVE. Failure to measure the lag is treated as permission to forge. On a chain paused across several epochs, the first live block then jumps over those epochs—the condition that the new IDLE mode and documented auto-off behavior are intended to prevent.

Evidence: Two temporary probes separately return false from waitForNodeTip() and null from the subsequent getNodeTip(). Both observe YanoRunMode.LIVE being started and startYanoOnly() returning true, despite never establishing that live production is safe.

Requested revision: Allow LIVE only after a successful tip read and an explicit safe-lag decision. On an unknown tip, retain IDLE if possible or fail startup with a retry message. Cover readiness timeout and a failed second tip read.

F3 [P1] Recheck wall-clock lag after a successful Yano-only backfill

Location: applications/cli/src/main/java/com/bloxbean/cardano/yacicli/localcluster/catchup/DevnetCatchUpService.java:494–501, consumed by lines 382–388 and 405–411.

Problem: Every non-null catch-up response unconditionally allows LIVE. A long backfill can succeed at its original target while wall clock has advanced by several epochs during the work. Companion mode pumps catch-up until it is sufficiently close to current time; Yano-only mode does not. Starting LIVE at this point skips the newly elapsed epochs.

Evidence: The pinned Yano v0.1.0-pre17 resolves to fb52ddc35b6bd0414064b5600b4144d2bd8d132f. Its catch-up implementation samples wall clock before producing the backfill and returns the resulting tip. A temporary DevKit probe uses 40-slot epochs, wall-clock slot 1,010 at request entry, and a successful response at slot 1,010 after the injected clock advances 200 seconds. DevKit makes exactly one catch-up call, prints success, and starts LIVE with the tip in epoch 25 and wall clock in epoch 30. This is a simulated slow response, not a claimed live-node timing measurement.

Requested revision: Compare the returned tip with fresh wall clock after success and repeat catch-up until the remaining gap is safe for handoff, with a bounded failure path if it cannot keep up. Account for the stop/restart interval too. Add a successful-but-slow-response regression alongside the existing failed-call clock-advance test.

Things to retain

• IDLE genuinely disables forging; CATCH_UP delays autonomous production while explicit backfill runs.
• Graceful node termination, immediate Yano process tracking, and cleanup of interrupted starts.
• The existing guard against starting LIVE after a failed cross-epoch backfill, plus the companion activeSlotsCoeff and multi-node restrictions.

Validation and scope

Executed from an isolated checkout of the reviewed SHA, with JAVA_HOME=/Users/satya/.sdkman/candidates/java/21.0.3-tem, in applications/cli:

./gradlew test --tests '*ChainLagTest' --tests '*DevnetCatchUpServiceTest' --tests '*RelaySyncWaiterTest' --tests '*YanoRunModeConfigTest' --tests '*YanoServiceStartTest' --tests '*ConsoleProgressTest' --tests '*YanoInstalledVersionTest' -x walletUiBuild -x walletUiInstall -x generateGitProperties --console=plain

./gradlew -I /tmp/yaci-pr196-review.y7KV6L/probes.gradle test --tests '*YanoServiceBackfillIntervalTest' --tests '*YanoServiceSlotLeaderTimeTravelTest' reviewProbeTest -x walletUiBuild -x walletUiInstall -x generateGitProperties --console=plain

Results: 58 existing tests passed; 0 failed, 0 errors, 0 skipped (48 in the first selection, 10 in the second). Four temporary probe tests passed, asserting the problematic observations described above. There is no separate YanoInstalledVersionTest class at this head; that filter added no tests. Probe sources and their Gradle init script were outside the checkout and were not committed.

The first test attempt failed in generateGitProperties because its JGit version could not resolve the linked worktree. Excluding that metadata task allowed the tests to run. Wallet UI build tasks were excluded because this review targets CLI code. The pinned Yano tag was independently verified through the GitHub API, then its source was read at the resolved commit. Live devnet/SDK end-to-end scenarios in the PR description were not rerun.

No implementation files were changed by this review. No approval is given at this SHA. Under the ADR process, any future design acceptance is not certification of implementation correctness or security.

[satran004]

Author reply: round r1 · head 8f92d08

r1 is pushed as 8f92d08 (on top of the reviewed 2d4dbe7, no force-push). I accepted all three findings after reading the cited paths: each one is reachable as described.

Finding	Change in r1	Where
F1 [P1] Propagate catch-up cancellation to the enclosing companion startup	startCluster now records its thread and a stopRequested flag (reset on entry; the body moved unchanged to doStartCluster). stopCluster sets the flag before cancelAndAwait, then waits up to 90 s for the start itself to return. Only then does it stop the process list, so anything the start registered is stopped. The start checks the flag before each process it would still create: the companion relay/producer startNode, the producer restart after the first-run handover, submit-api after catchUpOnRestart, and after yano-only startYanoOnly. When the flag is set, the start returns RunStatus(false, …) with "Start cancelled by a stop". A stop on the start's own thread does not wait, and a stop with no start running doesn't affect the next start.	ClusterStartService.startCluster, startCancelled, stopCluster, awaitStart
F2 [P1] Keep Yano non-forging when the auto-off tip check fails	startYanoOnlyWithoutCatchUp starts LIVE only after a successful IDLE start, a successful waitForNodeTip, a non-null tip, and ChainLag.liveStartSafe(). If the IDLE start or readiness check fails, or the tip is null, Yano is stopped and the start fails with "…not starting block production… Retry with 'start'". liveStartSafe() = !needsCatchUpOnStart() && epochsToCross() <= 1. The epoch term covers a securityParam set by hand that makes the window longer than an epoch, where a lag under ¾ of the window can still jump several boundaries.	DevnetCatchUpService.startYanoOnlyWithoutCatchUp, ChainLag.liveStartSafe, docs yano-node-modes.mdx (one sentence)
F3 [P1] Recheck wall-clock lag after a successful Yano-only backfill	backfillYanoOnly now loops, like companion: after each successful /catch-up it re-measures the lag from the returned tip against fresh wall clock, and stops only when the lag is ≤ ChainLag.yanoHandoffSlackSlots(). That slack is max(2, min(window/4, epochLength/2)), so the Yano stop/LIVE restart that follows cannot let the gap cross a second epoch boundary. It is bounded by MAX_PUMP_ROUNDS (20). A failed call, or running out of rounds, goes through one gaveUp rule (the same one as the existing failed-call guard): live start is allowed only when the remaining gap is inside one epoch; otherwise "…would skip N epochs. Retry with 'catch-up'." Block counts add up across rounds.	DevnetCatchUpService.backfillYanoOnly, gaveUp, ChainLag.yanoHandoffSlackSlots

Related fix found while doing F2 (not in the review): a yano-only devnet left IDLE with a lag between ¾ and 1 window could not be resumed: catch-up answered "The chain is live … Nothing to catch up", because that check only looks at lag.stalled(). F2 makes IDLE more common, so YanoService now records the run mode it started Yano in (runMode(), null when not running). ClusterStartService.catchUp treats a yano-only devnet whose Yano is not LIVE as needing a catch-up whatever the lag.

Deliberately unchanged: a companion restart whose catch-up fails for any reason other than a cancel still starts submit-api and reports success. The catch-up always leaves the Haskell producer running with the original topology, so the devnet is usable and catch-up can be retried. aFailedCatchUpWithoutAStopStillStartsTheDevnet pins this. If you think a non-cancel failure should also fail the start, say so and I'll change it.

Regression tests (new):

• ClusterStartServiceStopTest drives the real ClusterStartService.startCluster() (companion restart). Processes are fakes, and the catch-up collaborator is mocked to block until cancelled.
  • stopDuringTheRestartCatchUpAbortsTheStartAndWaitsForIt: start → catch-up → stopCluster on another thread. After the stop returns, the node launched before the stop has been stopped. No submit-api was started, nothing else was registered, the start returned stared=false, and isClusterRunning() is false.
  • aFailedCatchUpWithoutAStopStillStartsTheDevnet: a stop with nothing running first, then a start with a non-cancel failure, which still starts.
  • catchUpCommandRunsForAnIdleYanoOnlyDevnetInsideTheWindow / …IsANoOpForALiveYanoOnlyDevnet.
• DevnetCatchUpServiceTest:
  • F2: readiness timeout, null tip, and a short lag crossing 3 boundaries with a 210-slot window and 40-slot epochs. None of them starts LIVE.
  • F3: yanoOnlySlowBackfillIsRepeatedUntilItReachesTheCurrentWallClock, your scenario: 40-slot epochs, the first call returns slot 1,010 after the clock advanced 200 s. It now makes a second call and only then goes LIVE ("36 blocks to slot 1,210"). yanoOnlyBackfillThatNeverCatchesUpDoesNotStartLiveProduction: every call ends 200 s behind, the rounds run out, and the result is "would skip 5 epochs" with no LIVE.
• ChainLagTest: liveStartSafe and yanoHandoffSlackSlots.

Validation (JDK 21.0.3-tem, applications/cli):

• ./gradlew test -x walletUiBuild -x walletUiInstall: BUILD SUCCESSFUL, 87 tests, 0 failures, 0 errors, 0 skipped. That includes the 58 you ran (now 69 in those classes plus the new class).
• Mutation check: I put the 2d4dbe7 versions of ClusterStartService.java and DevnetCatchUpService.java back and kept the new tests. 7 tests failed: the two F1/idle ones, the three F2 ones, and the two F3 ones. The two control tests passed. The fixed files were then restored.
• I did not rerun live devnets or the SDK e2e suite for this round.

Over to you for round 2.