fix(health): detect stopped ingestion and stale chain data — liveness, freshness, backlog exposure (#186)#213
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bca6e8d95d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| bigname_storage::register_service_loop( | ||
| &pool, | ||
| bigname_storage::INDEXER_SERVICE_NAME, | ||
| &heartbeat_instance_id, | ||
| ) | ||
| .await?; |
There was a problem hiding this comment.
In the docker-compose.server.yml workflow I inspected, the indexer healthcheck starts after 20s and calls ensure_service_loop_heartbeat_recent, but this first heartbeat row is only registered after run_startup_bootstrap_backfills has fully returned. With active bootstrap ranges, a normal cold start can spend minutes or hours before reaching this line, so the container healthcheck and API /healthz report the indexer as not_started/unhealthy while it is actively bootstrapping rather than stopped. Register and refresh the loop heartbeat before or during bootstrap, or delay this healthcheck until live polling actually begins.
Useful? React with 👍 / 👎.
| let unlock = release_invalidation_apply_locks(&mut locks).await; | ||
| finish?; | ||
| unlock?; | ||
| record_loop_progress(pool, &mut loop_heartbeat).await?; |
There was a problem hiding this comment.
When a single projection invalidation takes longer than BIGNAME_WORKER_HEARTBEAT_MAX_AGE_SECS (20s by default; name_current applies are allowed to run up to 120s), the service heartbeat is only refreshed after the apply returns here, so the worker healthcheck and API /healthz can mark an actively working worker as stale. The existing claim heartbeat keeps the invalidation lease alive, but it does not update service_loop_heartbeats; refresh the service heartbeat during long applies or make the health threshold cover the longest normal apply.
Useful? React with 👍 / 👎.
| name_current::rebuild_name_current(pool, None), | ||
| requested_name_count | ||
| )); | ||
| record_loop_progress(pool, &mut loop_heartbeat).await?; |
There was a problem hiding this comment.
During automatic bootstrap replay, the heartbeat is recorded only after each full projection rebuild returns. On large datasets, a single name_current, record_inventory_current hydration, or address_names_current rebuild can exceed the default 20s worker heartbeat max age, so the worker healthcheck and API /healthz can mark the worker stale while it is doing normal replay work. Record progress inside the long rebuild/hydration loops or use a replay-specific health threshold.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
bigname/crates/storage/src/identity_facade/status.rs
Lines 99 to 102 in 469592c
Use the earliest pending status block
When both queues are present for a chain, for example a pending invalidation already tied to block 200 and a still-unscanned normalized-event change at block 100, this CASE reports latest_projected_block from the pending queue and never considers the older unscanned change. That under-reports /v1/status and /v2/status lag even though the public status contract describes unapplied normalized-event changes beyond the cursor as stale with lag fields identifying the affected chain; take the minimum first affected block across both sources instead.
AGENTS.md reference: AGENTS.md:L3-L3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| and leaves each omitted chain `unconfigured` and aggregate readiness | ||
| `degraded`. |
There was a problem hiding this comment.
When an expected chain is missing from BIGNAME_API_CHAIN_RPC_URLS but also has projection lag, the implementation does not leave aggregate readiness degraded: status_readiness returns Stale on projection_lag_blocks > 0 before checking network_head.status (apps/api/src/status_freshness.rs:135-139), and the added status tests assert network_head_status="unconfigured" with per-chain/root status="stale". Since these checked-in docs define the public semantics, this sentence will mislead operators and clients unless it is qualified as the no-projection-lag case or the code/tests are changed to match.
AGENTS.md reference: AGENTS.md:L3-L3
Useful? React with 👍 / 👎.
| let mut projection_derivation_started = false; | ||
| let projection_apply_generation = Arc::new(AtomicU64::new(0)); | ||
| let projection_apply_hydration_lock = Arc::new(Mutex::new(())); | ||
| let mut loop_heartbeat = heartbeat::LoopHeartbeat::new(heartbeat_instance_id, poll_interval); |
There was a problem hiding this comment.
When legacy reverse primary-name hydration is configured and a hydration pass takes longer than BIGNAME_WORKER_HEARTBEAT_MAX_AGE_SECS, this new main-loop heartbeat still goes stale: the spawned primary hydration loop receives no heartbeat/instance id and holds projection_apply_hydration_lock while it runs, so the main loop can block before it reaches the next record_if_due call. Pass heartbeat progress into that spawned hydration path or record a service-loop beat from it, otherwise /healthz and bigname-worker healthcheck can mark an actively hydrating worker stale.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d45de51197
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| ); | ||
| } | ||
| } | ||
| loop_heartbeat.record_if_due(&pool).await; |
There was a problem hiding this comment.
When BIGNAME_WORKER_CHAIN_RPC_URLS enables record-inventory text hydration, this beat is only recorded after the full post-bootstrap hydration pass returns. That pass scans/hydrates all eligible record_inventory_current rows and can easily run longer than the default BIGNAME_WORKER_HEARTBEAT_MAX_AGE_SECS (20s), so the worker healthcheck and /healthz can mark an actively hydrating worker stale even though the hydration helper now has a heartbeat-aware variant that records progress per page. Pass loop_heartbeat into this path instead of waiting until the end.
Useful? React with 👍 / 👎.
| .iter() | ||
| .map(|task| task.chain.clone()) | ||
| .collect::<Vec<_>>(); | ||
| startup_heartbeat.record(&pool, &live_chain_ids).await?; |
There was a problem hiding this comment.
When inline or auto startup has just finished the heartbeat-aware bootstrap, the post-bootstrap adapter sync branches above can still scan stored raw logs before this first live-scope beat is written. This is separate from the bootstrap-backfill case because this newly added beat still happens after sync_adapter_owned_raw_log_state/sync_discovery_adapter_owned_raw_log_state; with cold or large histories those syncs can exceed the 20s indexer heartbeat max age, so the indexer healthcheck and /healthz can mark an actively materializing indexer stale. Record startup heartbeat progress before or inside those syncs rather than only here.
Useful? React with 👍 / 👎.
d45de51 to
980d2f1
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 980d2f1ba8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| bigname_storage::record_service_loop_heartbeat( | ||
| pool, | ||
| bigname_storage::INDEXER_SERVICE_NAME, | ||
| heartbeat_instance_id, | ||
| &heartbeat_chains, |
There was a problem hiding this comment.
When inline adapter-sync mode sees a repository manifest/watch-plan change, this heartbeat is written only at the start of the poll tick, before the later sync_adapter_owned_raw_log_state call scans stored raw logs without any progress hook. On a large live manifest-refresh pass that normally takes longer than the 20s indexer heartbeat limit, the container healthcheck and API /healthz can mark an actively syncing indexer stale; pass heartbeat progress into that adapter-sync path or record during the scan.
Useful? React with 👍 / 👎.
| ) -> Result<ProjectionApplyIterationSummary> { | ||
| let derived = derive_once(pool).await?; | ||
| let applied = apply::apply_pending_invalidations( | ||
| loop_heartbeat.record_if_due(pool).await; |
There was a problem hiding this comment.
When the worker has a large backlog of normalized-event changes, derive_once can spend a full 5,000-change batch running the invalidation SQLs in one transaction before this newly added heartbeat is recorded. If that derive batch exceeds the default 20s worker heartbeat max age, /healthz and bigname-worker healthcheck report the worker stale even though it is actively enqueueing projection invalidations; record progress inside the derivation batch or before starting it.
Useful? React with 👍 / 👎.
|
Focused adversarial review of the accumulated heartbeat-threading + freshness logic (the load-bearing false-green/false-red correctness across all four rounds). CI is CLEAN 10/10, but the review found a P1 false-green the whole prior chain missed — fixing before merge. The heartbeat threading itself is clean — an exhaustive call-site audit found NO timer-driven or detached-task beat anywhere: every recurring beat requires completed work between firings (worker steady loop beats at iteration boundaries, invalidation applies beat after each drained group's finish/unlock, all seven rebuilds beat per completed task/tuple/batch with monolithic steps covered by named phases, hydration beats per completed RPC batch/upsert chunk, indexer poll beats per tick, backfill beats on progress-channel messages, discovery walk beats per admitted page). The r4 spawned-hydration fix is correct (shares the loop's LoopHeartbeat, idle iterations never beat, no phase-stomping, consistent lock order). Beat-write failure semantics match the ratified split (worker warn-and-continue-and-stay-due; indexer fatal). Freshness path pure/clamped/fail-closed. Startup threading (the live-deploy path) beats per page tied to progress. Rebase across #219/#221/#222 consistent. P1 (must-fix, being remediated): a stopped worker reports "running" on /healthz for up to the 43200s phase max-age via an ORPHANED phase row from a DEAD instance. A worker killed mid-named-phase (graceful ctrl_c included — the select! drops the replay future so finish_phase never runs; no shutdown deregistration exists) orphans its phase row; a replacement gets a new instance_id (HOSTNAME fallback) so register_service_loop reaps only its own rows; the read-ranking judges any phase-bearing instance alive purely by phase_age<=43200, so the DEAD instance's phase outranks a live-stale replacement → /healthz "ready" → release-smoke gates the deploy on that string, so the deploy gate passes with the worker stopped. Reproduced against the real preference SQL. The data model can't distinguish live-mid-phase from dead-mid-phase, so the window is inherent to the ranking, not an accident. Remediation (r5, in flight): register reaps other same-service instances' phase rows (single-writer worker topology), graceful-shutdown phase deletion, and a pinned per-service BIGNAME_HEARTBEAT_INSTANCE_ID — plus the repro test. P2 (false-red, r5): two live-poll adapter-refresh call sites (poll_loop.rs, discovery_refresh.rs) run un-heartbeated — the same class as the startup problem's live twin, with the _with_heartbeat plumbing already available; threading it in. P3s deferred to #224: un-instrumented startup families (reverse_claim + ens_v2_*); lag_blocks not clamped at 0 (pre-existing, cosmetic). Deploy note: the live stack runs the pre-#186 image (no heartbeats), so this P1 does not affect production today — it would only bite once #186 ships. |
|
r5's P1 false-green fix is fable-verified CLEAN (no P1/P2): the reap is boot-time-only before any phase can exist (call-site enumeration + the preserved live-mid-phase preference test), the write-fence is TOCTOU-free at the Postgres lock level (proven by a real lock-wait interleaving test), shutdown deregistration is genuine (tini forwards SIGTERM, awaited before exit; SIGKILL converges via the boot reap), and the P2 threading is progress-driven. Three self-converging documented P3s. But the connector found two more P2 false-reds on the r5 head — the same class, different call sites: the post-bootstrap adapter sync beats only after it returns ( Rather than another one-finding round, r6 is a comprehensive final sweep: fix both known gaps AND audit every worker/indexer long-op that can exceed the 20s stale window at production scale, classifying each as covered / newly-covered / cannot-exceed-with-reason / deferred-to-#224. Every added beat stays progress-tied (a wedged op stops beating → true-red), so no false-green is reintroduced. After this, remaining surface (if any) is explicitly enumerated in #224, not carried to r7. |
|
Minimal r6 (07d942f) — the two connector P2 false-reds — fable-verified CLEAN, no P1/P2. (The broad r6 "comprehensive audit" attempt exploded to 102 untested files; that's reset and tracked separately as #229; this round is a tight 3-file fix.)
Two P3s: batch-split efficiency (≤20 commits vs 1 — acceptable, The PR went DIRTY (#223 + #227 merged since its base, colliding on the shared API surface + regenerated openapi). Rebasing onto current main now (integrating #223's reserved-pool/load-shed-bypass additively with #213's loop-health/freshness payload; preserving #227's reconcile perf; regenerating openapi). Merging next after the rebase clears its gate. |
Persist per-instance loop heartbeats for indexer and worker health checks, and expose shared loop recency through /healthz. Refresh provider heads asynchronously behind timeout and cache bounds so v1 and v2 status report ingestion freshness, backlog counts, and dead letters without request-path RPC. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p
Split oversized replay checkpoint persistence along module seams and configure every e2e API process with its Anvil RPC mapping. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p
Thread heartbeat progress through post-bootstrap hydration and checkpointed adapter materialization, and align readiness precedence documentation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p
Reap orphaned scoped heartbeats on registration and deregister the worker on graceful shutdown with a process-row write fence. Pin stable compose identities while preserving live phase preference.\n\nThread live indexer manifest and discovery re-syncs through adapter page-progress heartbeats, with false-green and shutdown-race regressions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p
Keep the indexer heartbeat tied to post-bootstrap adapter page progress and split projection derivation into bounded committed progress units. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p
07d942f to
af82a55
Compare
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p
|
Final-head triage: the one connector comment (run.rs:150, "refresh heartbeat during post-bootstrap adapter sync") is a re-anchored stale finding, refuted by the code. It reads the plain function names Merging. #213 full tally: original fable+agy (found the P1 false-red healthz + heartbeat-stale P2s), r2 9-finding fix, r3 e2e-harness + file-size, r4 4-gap fix, r5 the P1 FALSE-GREEN fix (orphaned dead-instance phase — fable-verified clean, TOCTOU-free write-fence proven at the Postgres lock level), r6 minimal 2-gap fix (fable clean; the derive-batch split is crash-SAFER than base), rebase onto post-#223/#227 main (health handler integrates #223's reserved-pool + #213's loop-health additively, both on the reserved connection), clippy-fix (boxed the CLI variant). CI 10/10 CLEAN, connector 👍. Comprehensive long-op sweep tracked as #229; P3 follow-ups #224/#230. |
2 participants
Closes #186. A stopped or wedged service must look stopped; today every health surface stays green when ingestion halts.
Liveness — durable service-loop heartbeats
New
service_heartbeatsstorage keyed by (service, process instance, scope): the indexer writes one batched process + per-chain heartbeat per poll tick; the worker writes at loop ticks and actual progress boundaries between bootstrap projections and projection-apply units — no free-running timer can conceal a stuck operation. Containerhealthchecksubcommands inspect their own instance (BIGNAME_HEARTBEAT_INSTANCE_ID, defaulting to containerHOSTNAME), so a dead or wedged main loop fails its container check within the tunable window (default max age 20s vs the unchanged 30s compose cadence). Missing heartbeat = loop never started; old heartbeat = started then stopped/wedged./healthzuses the newest process heartbeat per service for replica-aware readiness.Freshness — provider-head comparison that can never block status
status_freshness: concurrent backgroundeth_blockNumberrefreshes (750ms provider timeout, 5s refresh interval, 30s cache TTL); status requests perform no provider I/O. A failed refresh reportsunavailableimmediately while retaining the last good head + lag as evidence. Readiness degrades to a distinct stale state past tunable block/seconds thresholds; seconds-freshness uses the additiveingestion_lag_seconds(v2's existinglag_secondsmeaning preserved).Exposure
/v1/statusand/v2/statusnow exposepending_invalidation_count,dead_letter_count,network_block,network_head_observed_at,network_head_age_seconds,network_head_status,ingestion_lag_blocks,ingestion_lag_seconds— the numbers previously folded into booleans. v2 envelope and existing fields unchanged (additive only).Frozen surfaces
/healthz types + frozen OpenAPI components, v1 status schema + regenerated artifact, v2 contract docs + ADR + envelope conformance, conformance harness/build wiring, env examples, deployment/storage/runbooks, smoke-script expectations — all updated coherently. No #157 readiness phases introduced (kept composable).
Verification
fmt/clippy/file-size; api 637 / indexer 509 / worker 264 passed; conformance 202; OpenAPI freeze/parity 6 + byte-exact artifact match; smoke scripts
bash -n. Codex$verify-loop: clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p