Conversation
ttlogan
left a comment
ttlogan
left a comment
There was a problem hiding this comment.
Checked the suspend/resume flow through the reconciler and teardown paths on a clean worktree. Ran the placement, api and teardown suites plus the onboarding/frontend ones: 195 passed. The two spots that matter are the lifecycle lease re-check in _launch and the fail-closed list() read; both are covered by tests (test_suspension_store_outage_blocks_placement, test_suspension_during_readiness_blocks_stale_plan). Looks good.
grafuls
left a comment
grafuls
left a comment
There was a problem hiding this comment.
One failure-path issue inline. The 128 targeted API, placement, lifecycle, and frontend tests pass; an additional active-provisioning/storage-failure reproduction exposes the ordering problem.
Persist manual teardown intent separately from node records and placement claims so automatic recovery cannot undo an operator's stop. Check the suspension before planning and again before launching a profile. List suspended hosts in the admin placement panel and provide an explicit resume action. Keep scheduled teardown and accidental-loss recovery unchanged, and document persistence and failure behavior. Cover restart persistence, real teardown, claim reset, concurrent changes, storage failures, authorization, and resume interactions. Closes quadsproject#160
b31b3c1 to
790c718
Compare
ttlogan
left a comment
ttlogan
left a comment
There was a problem hiding this comment.
The cancellation-ordering issue grafuls flagged is fixed on this head. It now inspects the active task first (active_provision), persists the suspension, then cancels only that inspected record via cancel_provision, which re-checks the identity. A store failure before cancellation leaves provisioning completely untouched and returns 503; the old cancel-then-write path is gone. Storage failure with an active provisioning task is now covered in test_teardown_lifecycle (the storage_fails parametrize), and the 44 suspension/teardown tests pass on the clean worktree. Looks good.
3 participants
Manual teardown now persistently suspends automatic placement for the host until an admin explicitly resumes it. Suspension survives node deletion, gateway restarts, and claim resets. Scheduled teardown and recovery from accidental registration loss retain their existing behavior.
The admin placement panel lists suspended hosts with a Resume action, and teardown confirmations explain the suspension. Resume restores eligibility subject to the usual availability, ownership, GPU qualification, exclusions, and retry rules.
Suspensions are stored separately from node records and placement claims. Teardown refuses to start if suspension cannot be saved, and a failed teardown leaves the host suspended. Revision checks prevent a stale resume from deleting newer teardown intent.
Validation: 2,848 full-suite tests passed, followed by additional regression checks covering persistence, real teardown, stale plans, claim reset, storage failures, authorization, and resume interactions. Combined coverage is 93.82%. Ruff, formatting, mypy, JavaScript syntax, and desktop/narrow-screen preview checks passed.
Closes #160.