Skip to content

fix: stop reporting restarts that never happened - #217

Merged
Shine-neko merged 4 commits into
mainfrom
fix/restart-count-terminal-marker
Aug 3, 2026
Merged

fix: stop reporting restarts that never happened#217
Shine-neko merged 4 commits into
mainfrom
fix/restart-count-terminal-marker

Conversation

@Shine-neko

Copy link
Copy Markdown
Contributor

restart_count no longer reports restarts that never happened.

The bug

apply_vm_start_failure assigned restart_count = MAX_RESTART_COUNT for every terminal error, using the counter as a "stop reconciling" marker rather than a tally.

So a deployment refused by host-memory admission — checked before anything is spawned — displayed:

$ ring deployment inspect <id>
Restart count : 5

Five restarts, for a workload where no process was ever created. Visible in the CLI, the API and the dashboard, sending anyone diagnosing it to look for an instability that never existed.

Why the marker was there, and why it is not needed everywhere

For statuses the scheduler still reconciles (ConfigError, ImagePullBackOff, CreateContainerError), exhausting the budget genuinely is what stops the retries — remove it and a permanent failure retries forever.

But a few terminal statuses are already excluded from the scheduler's filter by status alone (InsufficientResources, Failed, CrashLoopBackOff, Completed). For those the write bought nothing and only made the field lie.

scheduler_skips_by_status() now draws that line, and a test asserts it stays in sync with the scheduler's filter — a status added back to that filter without updating the helper would otherwise retry forever with no budget to stop it.

The same one-line change applies to containerd's handle_create_error, narrowed to InsufficientResources since its other terminal statuses are still reconciled.

Behaviour, verified on real microVMs

Scenario Before After
crash-loop (5 real boot attempts) 5 5
memory admission refusal (0 attempts) 5 0

The distinction the field is supposed to express is restored.

Also

tests/e2e/cloud-hypervisor/packer/*.log is now gitignored — a regenerated 1200-line serial console dump that kept getting swept into unrelated commits.

Testing

827 unit tests, clippy clean. Three existing classifier tests encoded the old behaviour and were updated, including every_start_failure_moves_the_restart_counter, whose invariant is now stated correctly: a failure must either move the counter or land outside the reconcile filter. Verified end to end with t14_crashloop (still 5) and t16_insufficient_memory (now 0).

@Shine-neko
Shine-neko merged commit c3331fa into main Aug 3, 2026
5 checks passed
@Shine-neko
Shine-neko deleted the fix/restart-count-terminal-marker branch August 3, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant