fix(worker): unify lifecycle into one authoritative, terminal-close() machine

[ChaoWao]

Scope

Narrowed, landable slice of P0.2. Folds the three overlapping lifecycle
truth-sources (_hierarchical_start_state str, _hierarchical_started bool,
_initialized bool) into one authoritative 5-state _Lifecycle
(NEW / INITIALIZING / READY / FAILED / CLOSED — no CLOSING) and gives
close() a semantically complete, single-shot terminal contract. The
concurrency- and journal-heavy pieces are explicitly deferred (list below) —
please do not read the code or tests as the final contract for those.

Delivered

• Single 5-state lifecycle guarded by _hierarchical_start_cv;
  _initialized / _hierarchical_started are read-only views. close() is a
  permanent commitment — publishes CLOSED atomically (the sole admission fence
  for the leased live-tree APIs) and never reverts to READY.
• close() while INITIALIZING fails fast. This worker does not cancel an
  in-progress init; the _cancel_requested wiring is removed (not dormant).
  A caller waits for READY or FAILED.
• Concurrent close() joins the in-flight _CloseAttempt it observed and
  sees that attempt's result; _CloseAttempt.done is set in a finally so a
  mid-teardown BaseException (KeyboardInterrupt) cannot strand joiners.
• Drain before teardown, on every attempt — a tree with a live leased op is
  never torn down. A drain-timeout leaves teardown un-attempted with the
  tree intact (the one retryable close() path) so a later close() completes
  it once the op drains.
• Teardown is single-shot and TERMINAL. Once it runs, an un-reclaimed
  resource (surviving child, failed C++ close, unfreed shm / domain / region /
  host-buffer / remote) leaks and is reported as an error; a later close()
  re-raises the same stored result and never re-drives a half-torn tree. A
  surviving child is now an error, so close() never returns success while a
  child is alive.
• Native teardown stays on the init-owner thread (device is
  aclrtSetDevice-bound); _ChipWorker.init / _Worker.init release the GIL.
• Startup eligible-target precheck factored into _eligible_target_need
  (per callable kind: LOCAL_PYTHON→SUB/next, LOCAL_CHIP→chip, REMOTE→its
  remote worker), raising at init() with namespace + hashid.

Explicitly NOT delivered — deferred to a follow-up

Do not read the code/tests below as the final contract.

1. register / unregister admission-safety vs a concurrent close().
   register publishes to the registry before it takes its lease, and
   unregister takes no lease — so both can still race close(). This PR
   does not claim they are fenced. Fixing it means leasing the whole
   READY gate → registry publication → native broadcast transaction (needs a
   deadlock-safe registry-lock / lifecycle-lock order).
2. Post-init dynamic register does not re-validate target eligibility
   (startup-only) — documented on register(). A chip callable dynamically
   registered on a chipless worker yields a non-dispatchable handle. Unifying the
   pre/post-init paths needs a device-free fake-chip-L3 harness; the ~10
   existing tests that register ChipCallables on chipless workers only prove
   the empty-broadcast facade, not a product expectation.
3. Cooperative init cancellation / shared INITIALIZING close transaction.
4. A genuinely retryable commit-barrier teardown journal — this PR ships
   terminal teardown, not a journal that drops a resource only after its
   native free succeeds.
5. Public ChipWorker ownership gate; recursive nested-shm journal.
6. add_worker() atomic child freeze (reads child lifecycle then locks only the
   parent, so a child can still init() concurrently after the check).
7. Startup rollback (_abort_hierarchical) may drop an un-reaped direct child.

Testing

• Device-free UT: 660 passed / 2 skipped.
• Simulation (a2a3sim): L2 vector_add, L3 multi_chip_dispatch, ST
  dynamic_register (5/5) — green.
• Onboard (a2a3/a5): full CI matrix green (st-onboard-a2a3,
  st-onboard-a5, ut-a2a3, ut-a5).

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 899b0961-8c7f-4ed9-b381-2e064a611d7f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Worker now uses an enum-based lifecycle with explicit startup cancellation, rollback, terminal closure, eligible-target validation, and improved hierarchical process cleanup. Tests were updated for the lifecycle model and expanded to cover concurrency, failure propagation, and terminal-state behavior.

Changes

Worker lifecycle and startup teardown

Layer / File(s)	Summary
Lifecycle contract and startup gating python/simpler/worker.py	Introduces _Lifecycle, lifecycle-derived properties, startup epoch coordination, eligible-target validation, and lifecycle-based registration gates.
Startup cancellation and teardown python/simpler/worker.py	Makes init() cancellation-aware, tracks startup process-group leaders, cleans orphaned descendants, and rewrites close() for bounded rollback and terminal closure.
Lifecycle concurrency and failure validation tests/ut/py/test_worker/test_startup_readiness.py	Tests concurrent initialization, active close cancellation, rollback backstops, target validation, terminal-state rejection, idempotent close, and shared failure propagation.
Lifecycle-based test scaffolding tests/ut/py/test_callable_identity.py, tests/ut/py/test_worker/test_host_buffer_registration.py, tests/ut/py/test_worker/test_host_worker.py, tests/ut/py/test_worker/test_l3_l2_message_queue.py, tests/ut/py/test_worker/test_l3_l2_orch_comm.py	Updates worker mocks and assertions to use _Lifecycle.READY and _Lifecycle.INITIALIZING, with related dispatch-target setup changes.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant Worker
  participant ChildWorkers
  participant ProcessGroups
  Caller->>Worker: init()
  Worker->>ChildWorkers: start and await readiness
  Caller->>Worker: close() during initialization
  Worker->>ChildWorkers: signal cancellation
  ChildWorkers-->>Worker: unwind startup
  Worker->>ProcessGroups: terminate remaining descendants
  Worker-->>Caller: raise cancellation and remain CLOSED

Loading

Possibly related PRs

• hw-native-sys/simpler#1381: Modifies the same hierarchical startup and lifecycle state handling.
• hw-native-sys/simpler#1396: Overlaps the readiness waiting, failure, and rollback paths.
• hw-native-sys/simpler#1397: Changes the same Worker.init() startup and lifecycle teardown behavior.

Poem

A rabbit watched the worker start,
Then close arrived with tidy art.
“Cancel,” it cried, “and clean the nest!”
The states grew clear, the tests did rest.
No orphaned process hopped away.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	It accurately summarizes the main change: consolidating worker lifecycle state into one authoritative, terminal close() machine.
Description check	✅ Passed	It describes the lifecycle unification, terminal close semantics, and related startup/rollback test coverage.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request refactors the worker lifecycle management by introducing a formal _Lifecycle enum state machine, replacing several ad-hoc state variables. It also implements active cooperative cancellation of in-progress initialization via close(), serializes rollback with a lock, cleans up orphaned grandchild processes, and adds a pre-init validation check to reject childless workers with pre-registered callables. The review feedback points out a critical race condition where _validate_eligible_targets() is called outside the _hierarchical_start_cv lock, which could allow a concurrent registration to bypass the check. Moving this validation inside the lock block is recommended to maintain safety invariants.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[ChaoWao]

Addressed the review in a22010d4:

1. Should-fix — wedged close() leaked the C++ scheduler. Fixed: the wedged backstop now calls self._cleanup_partial_init() (the same rollback entry init() uses) instead of bare _abort_hierarchical(), so _worker.close() runs before _worker is nulled — no leaked scheduler threads. _rollback_lock + the journal-cleared no-op keep double-invocation safe. test_l2_close_wedged_backstop_stays_closed now asserts the device is reclaimed (finalized == 1) and that a resuming init fails without resurrecting terminal CLOSED.

2. Consider — killpg leader-sweep PID reuse. Added a comment on the sweep acknowledging it shares the inherent killpg-reclaim assumption that the reaped leader's pid has not been reused as a new group leader (Linux allocates pids ~monotonically, so the window is negligible).

3. Gemini (high) — _validate_eligible_targets() race. Moved inside the with self._hierarchical_start_cv: block so a concurrent register() cannot install a target between the check and the epoch claim.

4. Acknowledged residual (nested-shm on hard-kill of a wedged-in-C++ grandchild). Correct to defer — needs a child→root channel; documented in the PR body and p0.2-design §G, not folded here.

CI: the st-sim-a2a3 (macos-latest) failure was TestBgemmHostBuildGraph::test_run → Golden mismatch on 'C': max_diff=nan — a numerical result in the bgemm host_build_graph compute path, which this diff does not touch (diff is worker.py lifecycle + worker tests only). st-sim-a5 (macOS+ubuntu), st-onboard-a2a3, and pre-commit all passed on the same commit. Re-running via this push to confirm it's flaky/platform, not a regression.

Local validation: device-free UT 643 passed / 2 skipped; sim a2a3sim L2/L3 green; ruff + pyright clean.

[ChaoWao]

Thanks for the deep review — it correctly caught that the machine linearized INITIALIZING but left READY-teardown, L2/remote mutation, and the cleanup budget unlinearized. Addressed in eae6fe25:

F1 — close() not atomic (fixed). Added a CLOSING state. close() claims READY → CLOSING atomically under the cv, so a racing run/register/create_host_buffer/second-close() observes the terminal transition (_initialized is READY-only) and cannot enter teardown. Teardown is extracted to _teardown_ready_tree() wrapped in a try/finally that always publishes CLOSED — a raising teardown step can no longer strand _lifecycle=READY. NEW/FAILED close clears the registry and commits CLOSED under the lock. New tests: test_concurrent_close_single_owner, test_l2_register_after_close_rejected.

F3 — L2 + remote mutation bypass (fixed). L2 register, _pre_start_unregister_if_needed (L2 branch), and the remote-dispatcher unregister early-branch all now go through _wait_out_init_locked (reject terminal, wait out INITIALIZING) — so a remote unregister during INITIALIZING waits for READY and actually sends the remote cleanup instead of dropping local state silently.

F5 — cleanup not end-to-end (fixed). _abort_hierarchical(deadline=…) now shares one budget across both phases, and the final reap is a bounded WNOHANG poll instead of a blocking waitpid(pid, 0) (a D-state survivor can no longer pin the rollback thread). Combined with F2, close() no longer chains its own grace after the init thread's.

F2 — wedged backstop unreachable/unsafe under the GIL (fixed by removing it). You're right: ChipWorker.init/_Worker.init don't release the GIL, so a real native hang blocks any second-thread close(), and cross-thread finalize() violates the same-thread C++ contract. close() no longer tears down cross-thread. On a wedged init it raises TimeoutError and leaves the worker INITIALIZING; the init thread self-rolls-back to FAILED same-thread when it unwedges (reclaiming its own device — no leak), and a retried close() reclaims the epoch. Rewrote the test to assert exactly that (no more GIL-releasing-fake interleaving), and removed the now-dead _rollback_lock.

F6a — original completion can fork (fixed). The CLOSING/CLOSED raise in _wait_out_init_locked now chains from self._startup_error, so a register-waiter that loses the wake race to close() still surfaces the original cause. (Your F6b was right — the L2 backstop no longer nulls _chip_worker cross-thread, so that assertion path is gone entirely with F2.)

F7 — eligible-target type-blind (fixed, with a correction). Now per-kind: a ChipCallable needs a chip device, a RemoteCallable a remote worker. But the "Python callable + device-only children → inert" case is not a defect — a Python callable on an L3 is the orchestrator that drives the chips (see my_orch), so it's valid with any child. Only ChipCallable-on-non-chip and truly-childless are inert. New test test_chip_callable_on_sub_only_l3_rejected.

Docs (fixed). Verified both stale against code and updated: callable-identity-registration.md (TASK_READY now fails loudly on an unprepared slot — worker.py:1402, no lazy-prepare) and remote-l3-worker-design.md (runner uses init() as the single startup point). Stale "starting" comment → INITIALIZING.

F4 — nested-shm leak on hard-kill (deferred, honestly). The remaining gap is real and matches P0.2 §1.8's recursive SHM journal: a next-level grandchild wedged in a C++ call and SIGKILLed leaks its own nested shm, and the scan-to-signal SIG_DFL window can default-kill a just-READY child before inner.close(). Closing this needs a parent↔child readiness-ACK handshake and a child→root shm-name channel (cross-process). I did not apply the "publish READY before restoring SIG_DFL" reorder — it doesn't fully close the window and it risks the load-bearing "a forked child must never unwind into the parent's start frames" invariant. Per P0.2 §6 I'm flagging this as the follow-up rather than folding a cross-process channel in here; the cooperative SIGTERM path remains leak-free for the common case.

Validation: device-free UT 646 passed / 2 skipped; ruff + pyright clean; sim a2a3sim L2 vector_add, L3 multi_chip_dispatch, and dynamic_register (5 cases) all pass.

[ChaoWao]

This round caught that my previous CLOSING addition introduced a critical revival bug — thank you. Addressed in f53daeb7; the PR is now honestly scoped as partial P0.2.

#1 (Critical — CLOSING revival, which I introduced). Fixed: init() now rejects CLOSING and CLOSED. I kept CLOSING (teardown needs _hierarchical_started for the host-buffer unmap broadcast) but closed the revival hole and added shared completion (below), which is the "at minimum, reject init on CLOSING" you called out. New test test_init_during_closing_is_rejected.

#3 (concurrent close doesn't share completion). Fixed: close() claims CLOSING and records _close_result; a joining close() waits out CLOSING and raises/returns the same result. _teardown_ready_tree is now error-accumulating (each step runs even if an earlier one raised; the first collected error is re-raised and stored).

#6 (cleanup deadline not end-to-end). Fixed: _cleanup_partial_init creates one absolute deadline and passes it to _abort_hierarchical(deadline=); removed the forced +1 s in the final reap (one guaranteed WNOHANG sweep, then poll until the shared deadline).

#7 (eligible-target — you were right, I had it backwards). Verified against the identity contract + _make_local_identity_tables + run() (an L3 orchestrator is invoked on the host, never registered). A registered Python callable is LOCAL_PYTHON, resolved only by a SUB/next loop — so it needs a sub/next/remote child, not a chip. Fixed the check and removed the incidental register(lambda) from the two chip-failure tests (the chip forks from device_ids regardless). New test test_registered_python_on_chip_only_l3_rejected.

Should-fix (all fixed). Remote frees now work during CLOSING teardown (_require_remote_worker_started accepts READY|CLOSING, before _worker.close() nulls it); add_worker rejects a non-NEW child; the _Lifecycle docstring documents CLOSING; the PR body/title are rewritten (no stale _rollback_lock, CLOSING documented, partial-P0.2 stated).

Not delivered — the P0.2 core that makes this partial (I am not marking P0.2 complete):

• Refactor AICPU core assignment to support dynamic block distribution #2 operation draining — admission control only; needs an active-operation lease/count that close() drains before teardown.
• Improve Type Safety: Upgrade AicpuExecute Parameter from void* to Runtime* #4 native thread-ownership / GIL — even a normal READY close() on a different thread than init() finalizes cross-thread, which ChipWorker/_Worker forbid. This needs an explicit decision — a same-thread-owner teardown contract, or cross-thread-safe native backends (C++/binding change). I've flagged it rather than pick a C++ path unilaterally (repo §6).
• Migrate AICore kernel compilation from C++ to Python and simplify DeviceRunner API #5 recursive nested-shm journal + scan-to-signal — needs a child→root shm-name channel + a readiness-ACK handshake.

I've listed all three in the PR body as follow-ups and will not claim P0.2 done. Happy to take direction on whether to pursue #2/#4/#5 in this PR or split them.

Validation: device-free UT 649 passed / 2 skipped; ruff + pyright clean; sim a2a3sim L2 vector_add + L3 multi_chip_dispatch pass.

[ChaoWao]

Addressed the most dangerous findings in 9f516f30; the rest are honestly tracked below.

Fixed this round:

• Multi-threaded AICPU Scheduler with Parallel Task Dispatch #1 (High) — lease timeout tore down live ops / reentrant close. close() now aborts without teardown if the drain doesn't complete in the budget (reports TimeoutError, leaves the worker usable), so it never frees the scheduler/_orch under a live op. A close() reentered from inside a leased operation (the orch-fn case) is rejected outright instead of draining its own lease. Tests: test_reentrant_close_from_operation_rejected, test_close_timeout_does_not_teardown.
• Refactor AICPU core assignment to support dynamic block distribution #2 (High) — public remote APIs during CLOSING. Reverted _require_remote_worker_started to READY-only; a separate internal _require_remote_transport (READY|CLOSING) is used only by the low-level _send_* helpers so close()'s teardown can still flush pending frees. Public remote_malloc/free/copy/import/... now reject during CLOSING.
• Migrate AICore kernel compilation from C++ to Python and simplify DeviceRunner API #5 (High/Med) — owner-dead cross-thread finalize. You're right that thread affinity (aclrtSetDevice) doesn't transfer. Removed the owner-dead exception: a READY close() is always rejected off the init-owner thread; tests that init on a transient thread now close on that owner thread. (Dedicated-owner-executor for true orphan reclaim is a separate follow-up.)
• Should-fix — lease fail-fast. _operation_lease no longer waits out INITIALIZING; it rejects a non-READY worker immediately, per the state/API matrix.

Still open (honestly not done — I don't want to under-claim):

• efactor: Rename Graph to Runtime to Better Reflect Its Responsibility #3 — complete the lease coverage: register/unregister, the L3 malloc/free/copy_to/copy_from, free_host_buffer, and the remote-memory APIs still hold only the admission gate, not a drain lease. (remote is now READY-gated, closing the CLOSING race, but not yet drained.)
• Improve Type Safety: Upgrade AicpuExecute Parameter from void* to Runtime* #4 — public ChipWorker (task_interface) has no owner/lifecycle guard; a dedicated cancellable-init seam for the high-level Worker is the cleaner fix.
• support extern func define in aicore #6 — _shutdown_child_group accumulates errors per group, not per PID/SHM, and clears the journal unconditionally.
• Simplify AicpuExecutor API and unify naming conventions #7 / add_worker TOCTOU / CLOSING-vs-roadmap — see the two contract decisions I'm raising with the author (eligible-target strictness, and roadmap §786 forbidding a CLOSING state vs the six-state machine here).

Validation: device-free UT 652 passed / 2 skipped, pyright 0, ruff clean; sim a2a3sim L2/L3 pass.

[ChaoWao]

Fixed the critical + clearly-correct findings in 80bde2be; the rest are honestly tracked in the (rewritten) PR body.

Fixed:

• Multi-threaded AICPU Scheduler with Parallel Task Dispatch #1 [critical] — retry tore down under a live op. close() now drains _active_ops on every attempt (not just the READY→CLOSED first one), so a retry while the op is still in flight drains again and defers, never freeing _worker/_chip_worker under a live op. New test test_close_retry_still_drains_before_teardown.
• Refactor AICPU core assignment to support dynamic block distribution #2 [high] — _CloseAttempt not pinned. A joiner now captures the exact attempt = self._close_completion, waits on that object, and replays its error (a retry installs a new attempt; joiners can't cross over). A failed close is now observed as a failure by its joiners.
• efactor: Rename Graph to Runtime to Better Reflect Its Responsibility #3 [high] — public APIs bypassing the lease. Lease-wrapped remote_export/remote_import and the post-start register broadcasts (chip/python/remote/L2), in addition to the earlier run/create_host_buffer/local-mem/remote-mem/free_host_buffer. Remaining: the unregister child-broadcast (its L2 path is under the registry lock — needs a lock-order-safe restructure) — flagged in the PR body, not silently dropped.
• Migrate AICore kernel compilation from C++ to Python and simplify DeviceRunner API #5 [high] — child shutdown could block forever + split journal. _shutdown_child_group now reaps with bounded WNOHANG to a shared deadline and treats each pid↔shm as one atomic pair (a shm is freed only once its pid is reaped, so a survivor keeps its control mailbox for retry).
• Improve Type Safety: Upgrade AicpuExecute Parameter from void* to Runtime* #4 [partial] — _has_live_resources now counts L3-L2 regions, CommDomains, host buffers, and pending remote frees/releases, so the CLOSED-incomplete/retry decision is accurate. The full per-resource commit-barrier journal (helpers that consume their journal before success) is still outstanding — flagged.

Still open (in the PR body): #4 full commit-barrier teardown journal; #6 (INITIALIZING-close into the shared transaction); the unregister broadcast lease; public-ChipWorker gate; add_worker freeze; the owner-death unreclaimable-by-design note (added); and the nested-shm channel (separate PR).

Validation: device-free UT 654 passed, pyright 0, ruff clean; sim a2a3sim L2/L3 pass.

[ChaoWao]

Narrowed the PR to a single-shot terminal close() per the review (retract the incomplete mechanisms rather than keep patching them). Pushed as `33630aa0`. Mapping to the last review:

1. Survivor child / PID·SHM journal misalignment — fixed: a child that does not reap within the budget is now an error, so close() never returns success with a live child. Under terminal teardown there is no re-zip/retry, so the min(len) re-pairing hazard is gone.
2. _CloseAttempt hang / concurrent retry — fixed: done is set in a finally (KeyboardInterrupt can't strand joiners); a joiner pins the exact attempt it observed; teardown-failure is terminal (no second attempt re-drives it).
3. INITIALIZING not a shared transaction — resolved by scoping down: close() now fails fast on INITIALIZING (no cancellation). The _cancel_requested wiring is removed, not dormant. Cooperative cancellation / shared transaction is a documented follow-up.
4. "Retryable teardown" not true — retracted. Teardown is now explicitly terminal: un-reclaimed region/domain/host-buffer/remote/child resources leak and are reported; no re-drive. Docstrings + PR body updated to stop claiming a retryable journal.
5. register/unregister admission race — claim retracted. The PR no longer states they are fenced against close() (register publishes before its lease; unregister takes none). Deferred with an explicit follow-up.
6. Startup rollback dropping an un-reaped direct child — acknowledged, not addressed this round; grouped with the commit-barrier reclaim follow-up (it's the _abort_hierarchical path, out of the close() narrowing).
7. Type-aware eligibility self-contradiction (dynamic register) — deferred by decision: enforcing it breaks ~10 device-free facade tests with no cheap fix, so this PR keeps eligibility startup-only and documents on register() that the dynamic path does not re-validate. Follow-up: fake-chip-L3 harness + unified pre/post-init check + LOCAL_CHIP pos/neg cases.

Deferred items are enumerated in the PR body. Device-free UT 653 passed / 2 skipped; a2a3sim L2/L3 + dynamic_register green. Watching CI on 33630aa0.

[ChaoWao]

Pushed 9190fe11 addressing the 5 must-fix items (all were real; #1/#2 were bugs I introduced in the narrowing):

1. Critical — drain-timeout joiner not pinned → concurrent double-teardown. Fixed: a caller that waits on an in-flight _CloseAttempt now always resolves against that attempt (raise its error / return) and never re-reads _close_completion or starts a retry. Only a fresh entry (that didn't wait) may retry a drain-timeout.
2. High — post-claim BaseException orphans completion. Fixed: the whole post-claim region (drain wait + teardown + registry clear) runs under one try/finally that always sets attempt.done. A KeyboardInterrupt mid-drain can no longer strand joiners. Completion is set before the registry clear.
3. High — operation lease not the sole admission point. Fixed: _require_remote_worker_started and _allocate_domain now check resource presence (_worker is not None), not public lifecycle. Since these run only inside an _operation_lease (which gated READY at acquire), an op admitted before a concurrent close() published CLOSED now completes during the drain instead of spuriously failing.
4. High — child non-zero exit treated as success. Fixed: _shutdown_child_group inspects WIFSIGNALED / WEXITSTATUS; an abnormal child exit (signal or non-zero code — failed chip finalize / raising inner_worker.close()) is now an error propagated to root close.
5. High — terminal residual silently succeeds. Fixed: close() synthesizes a terminal error naming the leaked resources when teardown ran but left a residual (_has_live_resources()), so it never returns success with a leak; a later close() replays the same result. _release_all_live_domains no longer deletes tracking on failure (keeps the leak detectable); _flush_pending_remote_frees already keeps its pending. New test test_close_terminal_residual_raises_and_replays.

Should-fix: stale "concurrent close cancels INITIALIZING" comments updated to the fail-fast contract (worker.py _StartupCancelled docstring, task_interface.cpp, worker_bind.h — GIL release retained, re-justified: it lets a concurrent close observe INITIALIZING and fail fast during a real native init). add_worker atomic child-freeze and the startup-rollback un-reaped-child (reviewer #6) are added to the deferred follow-up list.

Device-free UT 653 passed / 2 skipped; a2a3sim L2/L3 + dynamic_register green. (The st-onboard-a5 red on the prior push was a runner "Set up job" SSL/action-download infra flake — a2a3 passed on identical code; re-triggered by this push.)

[ChaoWao]

Pushed 98bdf384 closing the remaining findings (both reviews converged; all were real):

• [high] _CloseAttempt.done not all-path guaranteed + __del__-reentry self-deadlock. Fixed by construction: the claim and everything after it now run inside one try whose finally always completes the attempt (guarded by attempt is None for the pre-claim raises), so an async BaseException can't strand joiners. And on a terminal close the callable registries are detached under the lock but released after done is published and outside _registry_lock — a callable __del__ that reenters close() resolves against the done attempt instead of self-deadlocking. New test test_close_tolerates_callable_del_reentry. (Also covers reviewer A's narrow async-KI window.)
• [medium] terminal teardown not per-resource best-effort. Fixed: _cleanup_l3_l2_regions and _release_all_host_buffers now attempt every item (per-item try/finally) and raise the first error at the end, so one failing region/buffer no longer strands the rest.
• [medium] NEW/FAILED close didn't release the registry. Fixed: every terminal close detaches the callable/identity/handle registries (only a drain-timeout keeps them for the retry). New test test_close_of_registered_new_worker_releases_registry.
• [low] stale/over-promising source contract. Fixed: the _init_owner_thread comment ("any thread may cancel INITIALIZING"), the init() rollback docstring ("leaves no child … behind" → best-effort reap, un-reaped child deferred), and _teardown_ready_tree ("no register in flight" → register/unregister not yet lease-fenced) now match the delivered contract.
• [micro] _CloseAttempt.incomplete is an observable diagnostic (asserted by the drain-timeout tests) — kept.

Regression tests added for the two structural fixes (self-deadlock, NEW-worker registry release); the race-shaped cases (joiner-vs-drain-timeout, BaseException-mid-drain) are now correct by construction (single completion path).

Device-free UT 656 passed / 2 skipped; a2a3sim L2/L3 + dynamic_register green. Deferred follow-up list unchanged (register/unregister admission txn, dynamic eligibility + fake-chip-L3 harness, init cancellation, commit-barrier journal, add_worker freeze, startup-rollback child, ChipWorker gate, nested-shm).

[ChaoWao]

Pushed 09d49acf closing both blockers:

1. [high] close() completion still punch-through-able by KeyboardInterrupt. Fixed per the suggested ordering: the finally now completes the attempt first (publishes attempt.done before the blocking registry detach), then detaches. The registries are read by neither teardown nor _has_live_resources(), so this is free — it shrinks the strand window to the two non-blocking residual reads and no longer contains a lock acquire. retryable now keys off self._teardown_attempted (fact) instead of teardown_tree (intent). Regression test test_close_completion_published_before_registry_detach injects a KI in the detach and asserts done was already published + no later hang.

2. [high] serial child shutdown starves healthy later children. Fixed to two-phase: _broadcast_child_shutdown sends SHUTDOWN to every group first, then _reap_child_groups reaps all groups together with an interleaved poll over one shared deadline — a child wedged in one group can't burn the budget before the others are even SHUTDOWN. Regression test test_reap_child_groups_stuck_child_no_starvation (sub wedged, chip/next take a few polls to exit → healthy ones reaped, only the wedged child a reported survivor).

The "can never strand" comment is updated to state the precise remaining (non-blocking) window.

Device-free UT 658 passed / 2 skipped; a2a3sim L2/L3 (real-fork teardown via the new two-phase reap) + dynamic_register green. Deferred follow-up list unchanged.

[ChaoWao]

Pushed 745d37c7 closing the three blockers:

1. [high] completion still punch-through-able by KI. Restructured so completion is published in an innermost resilient finally whose only work is error/incomplete (plain assigns) then a minimal locked done=True; notify. Every fallible step (drain, teardown, residual synth, registry detach) runs before it and folds its error into the single result. Added a bounded joiner re-check (wait(timeout=_CLOSE_JOIN_RECHECK_S)) so a skipped notify (KI between done and notify) self-heals — the joiner re-observes done. The strand window is now the irreducible locked-publish itself; the code comment + PR wording no longer over-claim.
2. [high] completion-before-detach forked the result. Now the detach happens before completion, into a local under the lock, and a detach failure folds into result — so concurrent/later close()s never diverge (one success, one error). Old refs are still released after done + outside the lock (keeps the __del__-reentry fix). Rewrote the test (test_close_detach_interrupt_folds_into_single_result): registers a callable, teardown succeeds, injects a KI in the detach, and asserts the KI is the single folded result + a later close() replays it (never spurious success).
3. [high] reap deadline could expire before SHUTDOWN was sent. The child-reap grace is now computed after the SHUTDOWN broadcast (not at teardown entry), so a blocking pre-child cleanup step can't consume it. New test test_reap_deadline_starts_after_shutdown_broadcast.

Also updated the PR body Testing section (was stale: 653 / onboard-pending → 659 / full matrix green).

Device-free UT 659 passed / 2 skipped; a2a3sim L2/L3 (real-fork teardown) + dynamic_register green. Deferred follow-up list unchanged.

[ChaoWao]

Pushed b7a5fee3 closing the last strand window.

[high] done=True set after the CV acquire → KI during the acquire strands. Fixed exactly as suggested: the completion now sets attempt.error/incomplete/done as three plain attribute assigns and then acquires the CV only to notify_all(). done is published before the (interruptible, possibly-blocking) CV acquire, so a BaseException in the acquire or the notify can no longer leave done=False — and the bounded joiner re-check recovers the skipped notify. The only remaining window is an async exception landing between the error/incomplete and done plain assigns, which is irreducible in pure Python.

Added the requested deterministic regression test test_close_done_set_before_notify_lock_no_strand: it injects a KeyboardInterrupt on the completion's CV acquire (a wrapper that raises only once _close_completion.done is set) and asserts done was already published + a later close() does not hang.

Thanks for the earlier confirmations (detach-fold, SHUTDOWN-before-reap-grace, interleaved reap) — those stay as-is.

Device-free UT 660 passed / 2 skipped; a2a3sim L2/L3 + dynamic_register green. PR body Testing count synced to 660.

[ChaoWao]

Good catch — the test was self-consistent but not regression-catching. Fixed in 26218dad (test-only; worker.py unchanged):

• The KI is now injected on the completion's CV acquire by order (the 2nd acquire of a clean close), not gated on comp.done. So if the code regressed to set done inside that block, done stays False and the test fails.
• Asserts the injection actually fired (state["injected"] is True), that the first close() surfaced the KeyboardInterrupt, that _close_completion.done is True (the regression catch), and that the second close() resolves cleanly (_run_catch(...) is None) rather than hanging.
• Verified both directions: the test fails against a reverted (done-inside-lock) worker.py — assert ... and False (stranded) — and passes against the fix.

Device-free UT 660 passed / 2 skipped; full startup-readiness suite green.