Skip to content

feat(container-runner): report actors as crashed on unexpected platform SIGTERM - #5539

Merged
abcxff merged 7 commits into
mainfrom
stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk
Aug 11, 2026
Merged

feat(container-runner): report actors as crashed on unexpected platform SIGTERM#5539
abcxff merged 7 commits into
mainfrom
stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk

Conversation

@abcxff

@abcxff abcxff commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@claude

claude Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Review: report actors as crashed on unexpected platform SIGTERM

Summary: when the container-runner receives a platform SIGTERM (PLATFORM_RECLAIM, tracked by pre-existing code), it now walks a new ACTOR_CTXS registry (scc::HashMap<String, Ctx<GameServer>>) and calls ctx.stop_with_error(...) on every live actor before draining, so the reclaim shows up on the engine as a crash instead of a silent reallocation.

Findings

  1. Test coverage gap for the actual new behavior (medium).
    container-runner/examples/e2e-test/host/run-host-tests.sh Scenario 7 already SIGTERMs the runner and checks the child process is stopped ("sending SIGTERM to pid", no_orphan), but nothing asserts the thing this PR actually adds: that the actor is reported to the engine with an error/crash stop code rather than a normal stop. Given stop_with_error vs destroy differ specifically in StopCode, consider extending Scenario 7 (or adding a new scenario) to query the actor's status via the engine API after the SIGTERM and assert it reflects the crash/error path, not just that the child process died.

  2. Worth confirming: every SIGTERM is treated as an unexpected reclaim (medium, design question).
    crash_all_actors is gated only on the pre-existing PLATFORM_RECLAIM flag, which is set for any SIGTERM (as opposed to SIGINT). The hardcoded message ("likely OOM or running longer than 60 minutes") and this PR's crash-reporting both assume a SIGTERM always means an unexpected reclaim. The module doc-comment at the top of main.rs states the platform may also SIGTERM an idle instance as a normal reap path, and container platforms generally also send SIGTERM for benign reasons (rolling deploys of the runner image, node draining, capacity rebalancing) that aren't OOM or the 60-minute cap. If any of those can happen while actors are still live on the instance, this change will now report every live actor as crashed to the engine (potentially triggering crash-policy restarts/alerts) for what may be a benign shutdown. Worth confirming with whoever owns the hosting platform contract that a SIGTERM with live actors is always actually an unexpected reclaim on this platform.

  3. register_ctx's remove-then-insert isn't atomic (low).

async fn register_ctx(actor_id: &str, ctx: &Ctx<GameServer>) {
    ACTOR_CTXS.remove_async(actor_id).await;
    let _ = ACTOR_CTXS.insert_async(actor_id.to_string(), ctx.clone()).await;
}

Between the remove_async and insert_async there's a brief window where actor_id is absent from the map. If crash_all_actors runs in that exact window (a signal lands mid-registration), that actor is silently skipped and never reported as crashed. Given register_ctx is only called from on_start/duplicate-start handling, this window is narrow and probably fine in practice, but an atomic upsert (e.g. entry_async(...) + overwrite) would remove the race entirely rather than relying on timing.

  1. Minor: crash message duplicated three times.
    The string "...likely OOM or running longer than 60 minutes" now appears in spawn_signal_handler's two tracing::error! calls (main.rs) and again as the message passed to crash_all_actors. Not introduced fully by this diff (the log lines predate it), but this PR adds a third copy. A shared const would keep them from drifting if the wording ever changes.

What looks solid

  • ACTOR_CTXS uses scc::HashMap behind a LazyLock, consistent with the repo's "never Mutex<HashMap>" rule.
  • Registering the context only after a successful start (not before) correctly avoids leaking an entry for a generation whose start failed, since a failed start never runs on_destroy/on_sleep.
  • crash_all_actors runs before runtime.shutdown() while the envoy is still connected, which is necessary for the crash report to actually reach the engine, and this matters given the tight SIGTERM budget.
  • stop_with_error errors (e.g. a concurrent destroy already in flight) are caught and logged at debug, not error, which is appropriate since "destroy already requested" is an expected race rather than a bug.
  • Comment style matches CLAUDE.md conventions (complete sentences, no em dashes).

@abcxff
abcxff force-pushed the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch from 36380bb to d383183 Compare August 3, 2026 20:17
@abcxff
abcxff force-pushed the stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz branch from da132e5 to 94c1f83 Compare August 3, 2026 20:17
@abcxff
abcxff force-pushed the stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz branch from 94c1f83 to ae5019a Compare August 4, 2026 16:15
@abcxff
abcxff force-pushed the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch from d383183 to 2e35cdc Compare August 4, 2026 16:15
@abcxff
abcxff force-pushed the stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz branch from ae5019a to 4346f55 Compare August 5, 2026 17:09
@abcxff
abcxff force-pushed the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch from 2e35cdc to 3f1016b Compare August 5, 2026 17:09
@abcxff
abcxff force-pushed the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch from 3f1016b to 41fad8a Compare August 10, 2026 17:16
@abcxff
abcxff force-pushed the stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz branch from 4346f55 to 6733b2a Compare August 10, 2026 17:16
@abcxff
abcxff force-pushed the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch 2 times, most recently from ddda4b1 to a3dec62 Compare August 10, 2026 20:35
@abcxff
abcxff force-pushed the stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz branch from 69f80f4 to 9caf3ad Compare August 10, 2026 20:35
@abcxff
abcxff force-pushed the stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz branch from 9caf3ad to 823007e Compare August 10, 2026 22:09
@abcxff
abcxff force-pushed the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch from a3dec62 to 1b7821e Compare August 10, 2026 22:09
@abcxff
abcxff force-pushed the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch from 1b7821e to f623340 Compare August 11, 2026 13:39
@abcxff
abcxff force-pushed the stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz branch from 823007e to 776120d Compare August 11, 2026 13:39
@abcxff
abcxff force-pushed the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch from f623340 to 2adf6c7 Compare August 11, 2026 14:24
@abcxff
abcxff force-pushed the stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz branch from 776120d to 1804d76 Compare August 11, 2026 14:24
@abcxff
abcxff force-pushed the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch from 2adf6c7 to c78a0b7 Compare August 11, 2026 17:22
@abcxff
abcxff force-pushed the stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz branch from 1804d76 to 99da275 Compare August 11, 2026 17:22
@abcxff
abcxff changed the base branch from stack/feat-container-runner-log-unexpected-platform-sigterm-as-an-error-zsrurkrz to main August 11, 2026 17:23
@abcxff
abcxff merged commit c78a0b7 into main Aug 11, 2026
4 of 9 checks passed
@abcxff
abcxff deleted the stack/feat-container-runner-report-actors-as-crashed-on-unexpected-platform-sigterm-lzuymmyk branch August 11, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant