Add: eager, transactional, recursive Worker.init() for L3+#1397
Merged
ChaoWao merged 1 commit intoJul 19, 2026
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe worker lifecycle now performs eager recursive initialization during ChangesWorker startup lifecycle
Estimated code review effort: 5 (Critical) | ~90 minutes Sequence Diagram(s)sequenceDiagram
participant ParentWorker
participant ChildWorker
participant InnerWorker
participant ChipWorker
ParentWorker->>ChildWorker: fork during Worker.init()
ChildWorker->>InnerWorker: call inner_worker.init(deadline)
InnerWorker->>ChipWorker: fork and prepare startup callables
ChipWorker-->>InnerWorker: publish INIT_READY
InnerWorker-->>ChildWorker: publish subtree INIT_READY
ChildWorker-->>ParentWorker: report readiness
ParentWorker->>ParentWorker: register endpoints after local forks
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request transitions the hierarchical worker startup from lazy initialization on first run to eager and recursive initialization during Worker.init(). It introduces robust process group management and cooperative SIGTERM cancellation to ensure clean teardown of the entire process subtree upon failure or shutdown. Additionally, it prevents the corruption of device pointers during host-address rewriting and adapts the test harness to rehost host tensors into born-shared buffers eagerly. The review feedback highlights two critical robustness improvements: wrapping the initialization cleanup in a try...finally block to prevent hanging threads if cleanup fails, and safely defaulting to signal.SIG_DFL when restoring the SIGTERM handler to avoid a potential TypeError if the previous handler was None.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
ChaoWao
force-pushed
the
feat/eager-recursive-worker-init
branch
from
July 18, 2026 15:28
03dc2f1 to
4933364
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/simpler/worker.py`:
- Around line 3453-3461: Update the successful initialization block in init() so
_hierarchical_start_state is set to "started" for every worker level, including
level-2 workers; keep the existing _hierarchical_started assignment restricted
to level >= 3 and preserve the condition-variable notification.
In `@simpler_setup/scene_test.py`:
- Around line 247-250: Update the host-buffer cleanup around
self._worker.free_host_buffer(buf) to catch Exception instead of BaseException,
log the cleanup failure with relevant error details, and remove the silent pass.
Do not intercept process-control exceptions such as KeyboardInterrupt or
SystemExit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 427e9489-6c52-4c76-8b09-f641f48601eb
📒 Files selected for processing (16)
docs/hierarchical_level_runtime.mddocs/task-flow.mddocs/worker-manager.mdpython/bindings/task_interface.cpppython/simpler/remote_l3_session.pypython/simpler/remote_l3_worker.pypython/simpler/worker.pysimpler_setup/scene_test.pytests/st/a2a3/tensormap_and_ringbuffer/dynamic_register/test_dynamic_register.pytests/st/a2a3/tensormap_and_ringbuffer/test_l3_host_buffer_registration.pytests/ut/py/test_remote_l3_lifecycle.pytests/ut/py/test_scene_test_rehost.pytests/ut/py/test_worker/test_host_addr_rewrite.pytests/ut/py/test_worker/test_host_buffer_registration.pytests/ut/py/test_worker/test_host_worker.pytests/ut/py/test_worker/test_startup_readiness.py
ChaoWao
force-pushed
the
feat/eager-recursive-worker-init
branch
3 times, most recently
from
July 19, 2026 03:04
e90afbe to
6dbdf5a
Compare
Make Worker.init() the single, atomic startup point for level >= 3 hierarchies. init() now forks every local child, brings the whole subtree (recursively, for L4+) to READY, activates remote L3 sessions, starts the C++ scheduler, and only then commits READY -- or fails after a bounded rollback that leaves no child, mailbox, or session behind. run() / create_host_buffer() / the remote register/memory APIs no longer trigger startup. Core lifecycle (worker.py): - Atomic NEW -> INITIALIZING -> READY|FAILED commit under one lock; no observable started-but-not-initialized window. _cleanup_partial_init is the single rollback entry. - Recursion: a next-level child's init() is itself eager, so it publishes INIT_READY only after its own subtree is ready. - Fork-before-thread: remote add_remote_l3_socket (health thread) moved after the last local fork; sessions are only opened in _init_hierarchical. - Chip children upload their ChipCallables and prewarm before INIT_READY, removing the unbounded post-READY control_prepare from the startup path. - One startup deadline shared by all child groups and recursive descendants; a separate bounded grace covers cleanup. - Remove all five lazy _start_hierarchical() triggers. Cancellation domain (worker.py): - Each startup-root child is a process-group leader; descendants inherit the group. A mid-init child gets a cooperative SIGTERM that unwinds inner.init() and recursively reclaims its own grandchildren + nested shms; killpg is the hard subtree backstop, so no orphan is left to the resource_tracker. Remote path (remote_l3_session.py, remote_l3_worker.py): - Drop the runner's double-start; register inner callables into the eager snapshot; run the runner in its own session so the daemon reaps the whole L3->L2 subtree with a cooperative-SIGTERM-then-killpg cleanup. API linearization (worker.py): - register (local and remote), close, add_worker, and add_remote_worker all linearize against an in-progress epoch instead of racing it. SceneTest (simpler_setup/scene_test.py): - Add a transactional rehost adapter: post-init generate_args host tensors are moved into per-tensor born-shared create_host_buffer storage (dtype/shape/value preserved, non-contiguous rejected, LIFO release with partial-construction rollback) so the eagerly-forked children can see them. - create_host_buffer now accepts a sub-only L3 (chip OR sub child). - _rewrite_blob_host_addrs skips child_memory (device) tensors via a new binding-exported offset, so a device pointer inside a host range is not corrupted. Tests: - device-free coverage for the readiness barrier, eager contract, process-group subtree cancellation, INITIALIZING API races, the rehost adapter, the host-addr guard, and sub-only create_host_buffer. - Update two scene tests that relied on the removed lazy fork trigger. Docs: task-flow, worker-manager, and hierarchical lifecycle updated from lazy-on-first-run to eager.
This was referenced Jul 19, 2026
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 19, 2026
Post-merge hardening for hw-native-sys#1397 (eager init). Folds the three overlapping lifecycle sources (`_hierarchical_start_state`, `_hierarchical_started`, `_initialized`) into one `_Lifecycle` enum (NEW -> INITIALIZING -> READY | FAILED -> CLOSED) guarded by a single condition variable, and closes the gaps the split left: - CLOSED is terminal: close() no longer leaves a "started" epoch that a later init() could implicitly restart. - Every level claims the epoch at init() entry, fixing the L2 two-concurrent-init _chip_worker race (was level>=3 only). - close() actively cancels an in-progress init within a separate cleanup deadline instead of waiting out the startup budget; the readiness barrier polls the cancel token and unwinds through the one rollback path. - The READY commit sits inside the exception boundary and re-checks cancellation, so a cancelled-but-successful start rolls back instead of resurrecting a closed epoch (the L2 cancellation point). - Every waiter on a failed init observes the same original cause (immutable _startup_error), not a generic wrapper. - Root-visible process-group-leader journal plus a leader-reaped-but-descendants-alive killpg sweep reaps orphaned grandchildren on rollback. - _rollback_lock serializes the init thread's cleanup against close()'s wedged hard backstop; the FAILED transition is guarded so a wedged close's terminal CLOSED is never resurrected. - init() rejects a childless L3 that holds a pre-registered callable before allocating any startup resource. Tests: migrate white-box lifecycle fakes to `_Lifecycle`; add concurrent-L2 init, active-cancel, wedged-backstop, terminal-state, eligible-target, same-original-failure, and KeyboardInterrupt coverage in test_startup_readiness.py.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 19, 2026
… machine Post-merge hardening for hw-native-sys#1397 (eager init). Folds the three overlapping lifecycle sources (`_hierarchical_start_state`, `_hierarchical_started`, `_initialized`) into one `_Lifecycle` enum (NEW -> INITIALIZING -> READY | FAILED -> CLOSING -> CLOSED) guarded by a single condition variable, and linearizes every lifecycle transition: - CLOSED is terminal; a later init()/register never reopens the epoch. - Every level claims the epoch at init() entry, fixing the L2 two-concurrent-init _chip_worker race. - close() claims READY -> CLOSING atomically under the lock, so a racing run/register/create_host_buffer/second-close observes the terminal transition and cannot enter (or re-enter) teardown; teardown runs in a try/finally that always publishes CLOSED, so a failing step never strands the lifecycle in READY. - close() actively cancels an in-progress init within one cleanup budget. A wedged, GIL-holding native init cannot be torn down cross-thread (ChipWorker/_Worker require same-thread init/finalize), so close() reports TimeoutError rather than pretend; the init thread self-rolls-back to FAILED when it unwedges, and a retried close() reclaims the epoch. - L2 register/unregister and the remote-dispatcher unregister now route through the same lifecycle gate (reject a terminal worker, wait out an in-progress init) instead of bypassing it. - _abort_hierarchical shares one end-to-end deadline across both phases and reaps with a bounded WNOHANG poll, so a D-state survivor cannot pin the rollback thread past the cleanup budget. - Every waiter on a failed/closed init observes the same original cause. - Root-visible process-group-leader journal + a leader-reaped-but-descendants-alive killpg sweep reaps orphaned grandchildren on rollback. - init() rejects, per callable kind, a pre-registered callable with no matching dispatch target (a ChipCallable with no chip device, etc.) before allocating any startup resource. - Refresh stale docs/comments: TASK_READY now fails loudly on an unprepared slot (no lazy prepare); the remote runner uses init() as the single startup point. Tests: migrate white-box lifecycle fakes to `_Lifecycle`; add concurrent close, L2 register-after-close, wedged-close, terminal-state, type-aware eligible-target, same-original-failure, and KeyboardInterrupt coverage in test_startup_readiness.py.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 19, 2026
… machine Partial P0.2 (post-hw-native-sys#1397 hardening). Folds the three overlapping lifecycle sources (`_hierarchical_start_state`, `_hierarchical_started`, `_initialized`) into one `_Lifecycle` enum (NEW -> INITIALIZING -> READY | FAILED -> CLOSING -> CLOSED) guarded by a single condition variable, and linearizes lifecycle transitions: - CLOSING/CLOSED are terminal: a concurrent init() is rejected, never revives a closing epoch; every level claims the epoch at init() entry (fixing the L2 two-concurrent-init _chip_worker race). - close() claims READY -> CLOSING atomically under the lock (a racing run/register/create_host_buffer/second-close observes it and cannot enter teardown), tears down in an error-accumulating best-effort pass, and publishes CLOSED in a finally so a failing step never strands the lifecycle. Concurrent close()s join the same completion result. - close() actively cancels an in-progress init within one cleanup budget; a wedged, GIL-holding native init reports TimeoutError rather than being torn down cross-thread, and self-rolls-back to FAILED when it unwedges. - L2 register/unregister and the remote-dispatcher unregister route through the same lifecycle gate instead of bypassing it; the remote-free path is usable during CLOSING teardown. - _cleanup_partial_init / _abort_hierarchical share one end-to-end deadline and reap with a bounded WNOHANG poll. - init() rejects, per callable kind, a pre-registered callable with no matching dispatch target (a ChipCallable with no chip; a registered Python callable — LOCAL_PYTHON, resolved only by a SUB/next loop — with no sub/next/remote child) before allocating any startup resource. - add_worker rejects a non-NEW child; every waiter observes the same original startup cause; root-visible PGID journal + killpg sweep reaps orphaned grandchildren; docs/comments refreshed (TASK_READY no lazy prepare, remote runner uses init()). Explicitly NOT yet delivered (tracked as follow-ups; this PR does not complete P0.2): draining of in-flight admitted operations before teardown; same-thread-owner (or cross-thread-safe) native teardown; and the recursive nested-shm journal / scan-to-signal closure for hard-kill zero-residual. Tests: migrate white-box lifecycle fakes to `_Lifecycle`; add concurrent close, init-during-CLOSING, L2 register-after-close, wedged-close, add_worker-non-NEW, type-aware eligible-target, same-original-failure, and KeyboardInterrupt coverage.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 19, 2026
… machine Partial P0.2 (post-hw-native-sys#1397 lifecycle hardening). Folds the three overlapping lifecycle sources into one `_Lifecycle` enum (NEW -> INITIALIZING -> READY | FAILED -> CLOSING -> CLOSED) under a single condition variable and linearizes lifecycle transitions: - CLOSING/CLOSED are terminal: a concurrent init() is rejected, never revives a closing epoch; every level claims the epoch at init() entry (fixing the L2 two-concurrent-init race). - close() claims READY -> CLOSING atomically; a racing run/register/ create_host_buffer/second-close observes it and cannot enter teardown. Teardown is error-accumulating and a finally always publishes CLOSED, so a failing step never strands the lifecycle. Concurrent close()s join the same completion result. - Native thread-ownership contract: teardown of a READY tree runs only on the init-owner thread while it is alive (ChipWorker/_Worker init/finalize are same-thread-only); a foreign close() is rejected before touching the lifecycle, unless the owner thread has exited (orphaned tree, no concurrency). close() cancels an in-progress init within one cleanup budget and, if the native call is wedged, returns TimeoutError rather than tearing down cross-thread. _ChipWorker.init / _Worker.init release the GIL so a cancellation thread can enter Python; this does not permit cross-thread finalize. - L2 register/unregister and the remote-dispatcher unregister route through the same lifecycle gate; remote frees stay usable during CLOSING teardown. - One end-to-end cleanup deadline shared by _cleanup_partial_init / _abort_hierarchical; bounded WNOHANG reap. - init() rejects a childless L3 (no chip/sub/next/remote) that pre-registered a callable, before allocating any startup resource. add_worker rejects a non-NEW child; same original startup cause to all waiters; PGID journal + killpg sweep; docs/comments refreshed. Explicitly NOT yet delivered (this PR does not complete P0.2): draining of in-flight admitted operations before teardown; and the recursive nested-shm journal / scan-to-signal closure for hard-kill zero-residual. Tests: migrate white-box lifecycle fakes to `_Lifecycle`; add non-owner-close, owner+joiner, init-during-CLOSING, concurrent-init, wedged-close, add_worker-non-NEW, same-original-failure, and KeyboardInterrupt coverage.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 19, 2026
… machine Partial P0.2 (post-hw-native-sys#1397 lifecycle hardening). Folds the three overlapping lifecycle sources into one `_Lifecycle` enum (NEW -> INITIALIZING -> READY | FAILED -> CLOSING -> CLOSED) under a single condition variable and linearizes lifecycle transitions: - CLOSING/CLOSED are terminal: a concurrent init() is rejected, never revives a closing epoch; every level claims the epoch at init() entry (fixing the L2 two-concurrent-init race). - close() claims READY -> CLOSING atomically; a racing run/register/ create_host_buffer/second-close observes it and cannot enter teardown. Teardown is error-accumulating and a finally always publishes CLOSED, so a failing step never strands the lifecycle. Concurrent close()s join the same completion result. - Operation draining: run() and create_host_buffer() take a lease on the READY worker; close() claims CLOSING (blocking new leases) and drains the in-flight leases (bounded by the cleanup budget) before teardown, so an admitted operation is never mid-flight when the tree is torn down. - Native thread-ownership contract: teardown of a READY tree runs only on the init-owner thread while it is alive (ChipWorker/_Worker init/finalize are same-thread-only); a foreign close() is rejected before touching the lifecycle, unless the owner thread has exited (orphaned tree). close() cancels an in-progress init within one cleanup budget and, if the native call is wedged, returns TimeoutError rather than tearing down cross-thread. _ChipWorker.init / _Worker.init release the GIL so a cancellation thread can enter Python; this does not permit cross-thread finalize. - L2 register/unregister and the remote-dispatcher unregister route through the same lifecycle gate; remote frees stay usable during CLOSING teardown. - One end-to-end cleanup deadline shared by _cleanup_partial_init / _abort_hierarchical; bounded WNOHANG reap. init() rejects a childless L3 that pre-registered a callable. add_worker rejects a non-NEW child; same original startup cause to all waiters; PGID journal + killpg sweep; docs/comments refreshed. Still NOT delivered (this PR does not complete P0.2): operation-lease draining of register/unregister and the remote-memory APIs (run + create_host_buffer are leased); and the recursive nested-shm journal / scan-to-signal closure for hard-kill zero-residual. Tests: migrate white-box lifecycle fakes to `_Lifecycle`; add non-owner-close, owner+joiner, close-drains-operation, init-during-CLOSING, concurrent-init, wedged-close, add_worker-non-NEW, same-original-failure, KeyboardInterrupt.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 20, 2026
… machine Partial P0.2 (post-hw-native-sys#1397 lifecycle hardening). Folds the three overlapping lifecycle sources into one `_Lifecycle` enum (NEW -> INITIALIZING -> READY | FAILED -> CLOSING -> CLOSED) under a single condition variable and linearizes lifecycle transitions: - CLOSING/CLOSED are terminal: a concurrent init() is rejected, never revives a closing epoch; every level claims the epoch at init() entry. - close() claims READY -> CLOSING atomically; a racing run/register/buffer/ second-close observes it and cannot enter teardown. Teardown is error-accumulating; a finally always publishes CLOSED. Concurrent close()s join the same completion result. - Operation draining with hard safety: run() and create_host_buffer() take a fail-fast lease (READY-only, no wait). close() drains the leases before teardown; if they do NOT drain within the budget it aborts WITHOUT teardown (never frees the scheduler/_orch under a live op) and reports TimeoutError. A close() reentered from inside a leased operation (e.g. an orch fn) is rejected rather than draining its own lease. - Native thread-ownership: teardown of a READY tree runs only on the init-owner thread — always, since the device is bound to that thread (aclrtSetDevice) and affinity does not transfer even after it exits. close() cancels an in-progress init within one cleanup budget and returns TimeoutError if the native call is wedged, never tearing down cross-thread. _ChipWorker.init / _Worker.init release the GIL so a cancellation thread can enter Python; this does not permit cross-thread finalize. - Public remote-memory APIs are READY-only (reject during CLOSING); only the internal `_send_*` transport helpers accept CLOSING, so close()'s teardown can still flush pending remote frees while the sockets are up. - L2 register/unregister route through the lifecycle gate; one end-to-end cleanup deadline shared by _cleanup_partial_init / _abort_hierarchical; add_worker rejects a non-NEW child; same original cause to all waiters; PGID journal + killpg sweep; docs/comments refreshed. Still NOT delivered (this PR does not complete P0.2): operation-lease draining of register/unregister and the device/remote-memory APIs (run + create_host_buffer are leased); a public-ChipWorker ownership gate; per-resource (not per-group) teardown error accumulation; and the recursive nested-shm journal / scan-to-signal closure for hard-kill zero-residual. Tests: migrate white-box lifecycle fakes to `_Lifecycle`; add reentrant-close, close-timeout-no-teardown, non-owner-close, owner+joiner, close-drains-op, init-during-CLOSING, concurrent-init, wedged-close, and more.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 20, 2026
… machine Partial P0.2 (post-hw-native-sys#1397 lifecycle hardening). Folds the three overlapping lifecycle sources into one `_Lifecycle` enum (NEW -> INITIALIZING -> READY | FAILED -> CLOSING -> CLOSED) under a single condition variable and linearizes lifecycle transitions: - CLOSING/CLOSED are terminal: a concurrent init() is rejected, never revives a closing epoch; every level claims the epoch at init() entry. - close() claims READY -> CLOSING atomically; a racing run/register/buffer/ second-close observes it and cannot enter teardown. Teardown is error-accumulating; a finally always publishes CLOSED. Concurrent close()s join the same completion result. - Operation draining with hard safety: run() and create_host_buffer() take a fail-fast lease (READY-only, no wait). close() drains the leases before teardown; if they do NOT drain within the budget it aborts WITHOUT teardown (never frees the scheduler/_orch under a live op) and reports TimeoutError. A close() reentered from inside a leased operation (e.g. an orch fn) is rejected rather than draining its own lease. - Native thread-ownership: teardown of a READY tree runs only on the init-owner thread — always, since the device is bound to that thread (aclrtSetDevice) and affinity does not transfer even after it exits. close() cancels an in-progress init within one cleanup budget and returns TimeoutError if the native call is wedged, never tearing down cross-thread. _ChipWorker.init / _Worker.init release the GIL so a cancellation thread can enter Python; this does not permit cross-thread finalize. - Public remote-memory APIs are READY-only (reject during CLOSING); only the internal `_send_*` transport helpers accept CLOSING, so close()'s teardown can still flush pending remote frees while the sockets are up. - L2 register/unregister route through the lifecycle gate; one end-to-end cleanup deadline shared by _cleanup_partial_init / _abort_hierarchical; add_worker rejects a non-NEW child; same original cause to all waiters; PGID journal + killpg sweep; docs/comments refreshed. Still NOT delivered (this PR does not complete P0.2): operation-lease draining of register/unregister and the device/remote-memory APIs (run + create_host_buffer are leased); a public-ChipWorker ownership gate; per-resource (not per-group) teardown error accumulation; and the recursive nested-shm journal / scan-to-signal closure for hard-kill zero-residual. Tests: migrate white-box lifecycle fakes to `_Lifecycle`; add reentrant-close, close-timeout-no-teardown, non-owner-close, owner+joiner, close-drains-op, init-during-CLOSING, concurrent-init, wedged-close, and more.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 20, 2026
… machine Partial P0.2 (post-hw-native-sys#1397 lifecycle hardening). One authoritative *public admission* lifecycle — 5 states, NEW -> INITIALIZING -> READY | FAILED -> CLOSED — guarded by a single condition variable. "Closing in progress" is a private per-attempt teardown phase, never a public state. - Admission is decided solely by lifecycle: CLOSED rejects every public live-tree API, permanently. close() is a commitment, not a reversible attempt — it publishes CLOSED at claim and never reverts to READY. - Per-attempt _CloseAttempt: close() installs a fresh attempt at claim; concurrent close()s pin to the attempt they observe and see the same outcome (fixing the single-slot joiner hazard). A drain-timeout or teardown failure leaves the worker CLOSED with the attempt INCOMPLETE (retryable); a later close() re-drives teardown on the owner thread. A tree with a live operation is never torn down under it. - Teardown drives children off *resource presence* (_worker / mailboxes), never off lifecycle: the _require_remote_transport (private, resource-gated) vs _require_remote_worker_started (public READY-only) split, extended to the host-buffer unmap broadcast; per-PID/SHM error accumulation that leaves an un-reclaimed resource journaled for a later close() to retry. - Operation-lease draining (run + create_host_buffer today) with reentrancy rejection and fail-fast admission (READY-only, no wait). - Native teardown runs only on the init-owner thread (device-bound), always; init GIL-released so a cancellation thread can enter Python. - Eligible-target precheck is per callable kind, keyed on the same target_namespace -> child-loop map _make_local_identity_tables applies (LOCAL_PYTHON -> SUB/next; LOCAL_CHIP -> chip; REMOTE -> its remote worker), raising at init with namespace + hashid; the dynamic_register capacity ST gives its Python fillers a SUB resolver. Still NOT delivered (this PR does not complete P0.2): operation-lease draining of register/unregister, the device/remote-memory APIs, and free_host_buffer; a public-ChipWorker ownership gate; the recursive nested-shm journal / scan-to-signal closure for hard-kill zero-residual. Tests: 5-state + per-attempt completion + resource-gated teardown coverage (reentrant-close, close-timeout-defers+retry, owner+joiner, non-owner-close, init-after-close-claim, mapping-derived eligible-target, ...).
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 20, 2026
… machine Partial P0.2 (post-hw-native-sys#1397 lifecycle hardening). One authoritative *public admission* lifecycle — 5 states, NEW -> INITIALIZING -> READY | FAILED -> CLOSED — guarded by a single condition variable. "Closing in progress" is a private per-attempt teardown phase, never a public state. - Admission is decided solely by lifecycle: CLOSED rejects every public live-tree API, permanently. close() is a commitment, not a reversible attempt — it publishes CLOSED at claim and never reverts to READY. - Per-attempt _CloseAttempt: close() installs a fresh attempt at claim; concurrent close()s pin to the attempt they observe and see the same outcome (fixing the single-slot joiner hazard). A drain-timeout or teardown failure leaves the worker CLOSED with the attempt INCOMPLETE (retryable); a later close() re-drives teardown on the owner thread. A tree with a live operation is never torn down under it. - Teardown drives children off *resource presence* (_worker / mailboxes), never off lifecycle: the _require_remote_transport (private, resource-gated) vs _require_remote_worker_started (public READY-only) split, extended to the host-buffer unmap broadcast; per-PID/SHM error accumulation that leaves an un-reclaimed resource journaled for a later close() to retry. - Operation-lease draining (run + create_host_buffer today) with reentrancy rejection and fail-fast admission (READY-only, no wait). - Native teardown runs only on the init-owner thread (device-bound), always; init GIL-released so a cancellation thread can enter Python. - Eligible-target precheck is per callable kind, keyed on the same target_namespace -> child-loop map _make_local_identity_tables applies (LOCAL_PYTHON -> SUB/next; LOCAL_CHIP -> chip; REMOTE -> its remote worker), raising at init with namespace + hashid; the dynamic_register capacity ST gives its Python fillers a SUB resolver. Still NOT delivered (this PR does not complete P0.2): operation-lease draining of register/unregister, the device/remote-memory APIs, and free_host_buffer; a public-ChipWorker ownership gate; the recursive nested-shm journal / scan-to-signal closure for hard-kill zero-residual. Tests: 5-state + per-attempt completion + resource-gated teardown coverage (reentrant-close, close-timeout-defers+retry, owner+joiner, non-owner-close, init-after-close-claim, mapping-derived eligible-target, ...).
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 20, 2026
… machine Partial P0.2 (post-hw-native-sys#1397 lifecycle hardening). One authoritative *public admission* lifecycle — 5 states, NEW -> INITIALIZING -> READY | FAILED -> CLOSED — guarded by a single condition variable. "Closing in progress" is a private per-attempt teardown phase, never a public state. - Admission is decided solely by lifecycle: CLOSED rejects every public live-tree API, permanently. close() is a commitment, not a reversible attempt — it publishes CLOSED at claim and never reverts to READY. - Per-attempt _CloseAttempt: close() installs a fresh attempt at claim; concurrent close()s pin to the attempt they observe and see the same outcome (fixing the single-slot joiner hazard). A drain-timeout or teardown failure leaves the worker CLOSED with the attempt INCOMPLETE (retryable); a later close() re-drives teardown on the owner thread. A tree with a live operation is never torn down under it. - Teardown drives children off *resource presence* (_worker / mailboxes), never off lifecycle: the _require_remote_transport (private, resource-gated) vs _require_remote_worker_started (public READY-only) split, extended to the host-buffer unmap broadcast; per-PID/SHM error accumulation that leaves an un-reclaimed resource journaled for a later close() to retry. - Operation-lease draining (run + create_host_buffer today) with reentrancy rejection and fail-fast admission (READY-only, no wait). - Native teardown runs only on the init-owner thread (device-bound), always; init GIL-released so a cancellation thread can enter Python. - Eligible-target precheck is per callable kind, keyed on the same target_namespace -> child-loop map _make_local_identity_tables applies (LOCAL_PYTHON -> SUB/next; LOCAL_CHIP -> chip; REMOTE -> its remote worker), raising at init with namespace + hashid; the dynamic_register capacity ST gives its Python fillers a SUB resolver. Still NOT delivered (this PR does not complete P0.2): operation-lease draining of register/unregister, the device/remote-memory APIs, and free_host_buffer; a public-ChipWorker ownership gate; the recursive nested-shm journal / scan-to-signal closure for hard-kill zero-residual. Tests: 5-state + per-attempt completion + resource-gated teardown coverage (reentrant-close, close-timeout-defers+retry, owner+joiner, non-owner-close, init-after-close-claim, mapping-derived eligible-target, ...).
This was referenced Jul 21, 2026
4 tasks
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…rker A child (kind4) device pointer is a bare int today — no owner, no generation (tensor.h child_memory is one bit). Once such a pointer entered a TaskArgs it could be freed, copied, or run on a worker that never allocated it: a raw device VA is not globally unique, so worker B silently touched worker A's (or illegal) device memory. Guard ① (hw-native-sys#1397) only stopped host-addr rewrite from corrupting a device VA; nothing checked the pointer's owning worker. Track live child pointers by their exact (worker_id, ptr) provenance and validate every consumer (kind4 device ops and dispatch; L2 Worker.run is not a consumer): - Provenance table on the Worker: a typed (worker_id, ptr) -> entry map (malloc_owned + domain_allocation_ids), so a malloc base and an aliasing CommDomain buffer are distinct roles. Recorded only after the backend op succeeds; check -> native op -> update is linearized under one private lock. Cleared on close(). - malloc/free/copy_to/copy_from: recorded at malloc; free requires an exact live malloc base (a domain/interior/stale/wrong-worker pointer is rejected) and keeps provenance if the native free fails (retryable); copy requires the device-side pointer to be live on that worker. Covers both the Worker.malloc (L2 + L3) and the direct orch.malloc/copy/free paths, which share the single Orchestrator choke. - submit_next_level / submit_next_level_group: a child_memory argument must resolve to exactly one eligible target worker (judged on the resolved eligibility, not a raw worker=-1) and be a live allocation there; 0 or >= 2 candidates is ambiguous and rejected. The resolved owner is then passed to C++ as the effective affinity so the child TensorKey is keyed by its owner, not the raw -1 — otherwise the same buffer submitted once as -1 and once as W gets two keys and its dependency is missed (per member for the group path). - CommDomain window / buffer pointers enter provenance on allocate_domain and are revoked in _release_domain_now BEFORE the backend free (a commit barrier): once release begins the pointers are undispatchable, so a concurrent dispatch cannot validate a being-freed pointer and a partial backend-release failure leaves them dropped rather than "live forever". Boundary: catches stale-before-reuse, NOT strict ABA (a re-malloc of the same VA becomes live again) — that needs P1 generation handles. Python-only, no wire / C++ / ABI change; the raw-C++ deep-bypass (worker._orch._o.malloc) is out of scope. Adds 36 device-free UT (provenance table, target resolution, orchestrator memory ops incl. native-error transactions, single/group dispatch + resolved affinity, CommDomain dispatch and release-before-free commit barrier, L2 path).
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…rker A child (kind4) device pointer is a bare int today — no owner, no generation (tensor.h child_memory is one bit). Once such a pointer entered a TaskArgs it could be freed, copied, or run on a worker that never allocated it: a raw device VA is not globally unique, so worker B silently touched worker A's (or illegal) device memory. Guard ① (hw-native-sys#1397) only stopped host-addr rewrite from corrupting a device VA; nothing checked the pointer's owning worker. Track live child pointers by their exact (worker_id, ptr) provenance and validate every kind4 consumer (device ops and dispatch; L2 Worker.run is not a consumer): - Provenance table on the Worker: a typed (worker_id, ptr) -> entry map (malloc_owned + domain_allocation_ids), so a malloc base and an aliasing CommDomain buffer are distinct roles. check -> native op -> table update is linearized under one private lock. Cleared on close(). - malloc: recorded after the backend malloc succeeds. free: a safety-first commit barrier — validate an exact live malloc base, then revoke provenance BEFORE the native free, so an async unwind (e.g. a KeyboardInterrupt after the binding returns) can never leave a freed address live; a native-free failure becomes a terminal leak, never a re-authorized maybe-freed address. copy_* require the device-side pointer live on that worker. Covers both Worker.malloc (L2 + L3) and the direct orch.malloc/copy/free paths (single Orchestrator choke), L2 and L3 unified. - submit_next_level / submit_next_level_group: a child_memory argument must resolve to exactly one eligible target worker (judged on the resolved eligibility, not a raw worker=-1) and be a live allocation there; 0 or >= 2 candidates is ambiguous and rejected. The resolved owner is passed to C++ as the effective affinity so the child TensorKey is keyed by its owner, not the raw -1 (otherwise the same buffer submitted once as -1 and once as W gets two keys and its dep is missed). The group path materialises a full per-member affinity only when workers is empty/None; a non-empty workers list is length- checked against args (never silently padded, which would bypass the C++ length check). - CommDomain window / buffer pointers enter provenance on allocate_domain and are revoked in _release_domain_now BEFORE the backend free (a commit barrier): the deferred window stays dispatchable, but once physical release begins the pointers are undispatchable, so a concurrent dispatch cannot validate a being-freed pointer and a partial backend-release failure leaves them dropped rather than "live forever". Boundary: catches stale-before-reuse, NOT strict ABA (a re-malloc of the same VA becomes live again) — that needs P1 generation handles. Python-only, no wire / C++ / ABI change; the raw-C++ deep-bypass (worker._orch._o.malloc) is out of scope. Adds 39 device-free UT (provenance table, target resolution, orchestrator memory ops incl. free commit-barrier + native-error transactions, single/group dispatch + resolved affinity + group length validation, CommDomain dispatch and release-before-free commit barrier, L2 path).
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…rker A child (kind4) device pointer is a bare int today — no owner, no generation (tensor.h child_memory is one bit). Once such a pointer entered a TaskArgs it could be freed, copied, or run on a worker that never allocated it: a raw device VA is not globally unique, so worker B silently touched worker A's (or illegal) device memory. Guard ① (hw-native-sys#1397) only stopped host-addr rewrite from corrupting a device VA; nothing checked the pointer's owning worker. Track live child pointers by their exact (worker_id, ptr) provenance and validate every kind4 consumer (device ops and dispatch; L2 Worker.run is not a consumer): - Provenance table on the Worker: a typed (worker_id, ptr) -> entry map (malloc_owned + domain_allocation_ids), so a malloc base and an aliasing CommDomain buffer are distinct roles. check -> native op -> table update is linearized under one private lock. Cleared on close(). - malloc: recorded after the backend malloc succeeds. free: a safety-first commit barrier — validate an exact live malloc base, then revoke provenance BEFORE the native free, so an async unwind (e.g. a KeyboardInterrupt after the binding returns) can never leave a freed address live; a native-free failure becomes a terminal leak, never a re-authorized maybe-freed address. copy_* require the device-side pointer live on that worker. Covers both Worker.malloc (L2 + L3) and the direct orch.malloc/copy/free paths (single Orchestrator choke), L2 and L3 unified. - submit_next_level / submit_next_level_group: a child_memory argument must resolve to exactly one eligible target worker (judged on the resolved eligibility, not a raw worker=-1) and be a live allocation there; 0 or >= 2 candidates is ambiguous and rejected. The resolved owner is passed to C++ as the effective affinity so the child TensorKey is keyed by its owner, not the raw -1 (otherwise the same buffer submitted once as -1 and once as W gets two keys and its dep is missed). The group path materialises a full per-member affinity only when workers is empty/None; a non-empty workers list is length- checked against args (never silently padded, which would bypass the C++ length check). A group must dispatch to distinct workers, so two members that resolve to the same owner (a duplicate non-negative affinity) are rejected rather than silently serialized on one WorkerThread. - CommDomain window / buffer pointers enter provenance on allocate_domain and are revoked in _release_domain_now BEFORE the backend free (a commit barrier): the deferred window stays dispatchable, but once physical release begins the pointers are undispatchable, so a concurrent dispatch cannot validate a being-freed pointer and a partial backend-release failure leaves them dropped rather than "live forever". Boundary: catches stale-before-reuse, NOT strict ABA (a re-malloc of the same VA becomes live again) — that needs P1 generation handles. Python-only, no wire / C++ / ABI change; the raw-C++ deep-bypass (worker._orch._o.malloc) is out of scope. Adds 40 device-free UT (provenance table, target resolution, orchestrator memory ops incl. free commit-barrier + native-error transactions, single/group dispatch + resolved affinity + group length/duplicate-worker validation, CommDomain dispatch and release-before-free commit barrier, L2 path).
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…rker A child (kind4) device pointer is a bare int today — no owner, no generation (tensor.h child_memory is one bit). Once such a pointer entered a TaskArgs it could be freed, copied, or run on a worker that never allocated it: a raw device VA is not globally unique, so worker B silently touched worker A's (or illegal) device memory. Guard ① (hw-native-sys#1397) only stopped host-addr rewrite from corrupting a device VA; nothing checked the pointer's owning worker. Track live child pointers by their exact (worker_id, ptr) provenance and validate every kind4 consumer (device ops and dispatch; L2 Worker.run is not a consumer): - Provenance table on the Worker: a typed (worker_id, ptr) -> entry map (malloc_owned + domain_allocation_ids), so a malloc base and an aliasing CommDomain buffer are distinct roles. Each op is atomic under one private lock; ordering is safety-first (record after alloc, revoke before free). Async interruption is fail-closed: a role is fully set before its entry is inserted, the last role is deleted directly (no role-less entry ever exists), and every live check tests the roles, not key presence, so an entry momentarily left empty never re-authorizes a freed pointer. Cleared on close(). - malloc: recorded after the backend malloc succeeds. free: a safety-first commit barrier — validate an exact live malloc base, then revoke provenance BEFORE the native free, so an async unwind (e.g. a KeyboardInterrupt after the binding returns) can never leave a freed address live; a native-free failure becomes a terminal leak, never a re-authorized maybe-freed address. copy_* require the device-side pointer live on that worker. Covers both Worker.malloc (L2 + L3) and the direct orch.malloc/copy/free paths (single Orchestrator choke), L2 and L3 unified. - submit_next_level / submit_next_level_group: a child_memory argument must resolve to exactly one eligible target worker (judged on the resolved eligibility, not a raw worker=-1) and be a live allocation there; 0 or >= 2 candidates is ambiguous and rejected. The resolved owner is passed to C++ as the effective affinity so the child TensorKey is keyed by its owner, not the raw -1 (otherwise the same buffer submitted once as -1 and once as W gets two keys and its dep is missed). The group path materialises a full per-member affinity only when workers is empty/None; a non-empty workers list is length- checked against args (never silently padded, which would bypass the C++ length check). A group must dispatch to distinct workers, so two members that resolve to the same owner (a duplicate non-negative affinity) are rejected rather than silently serialized on one WorkerThread. - CommDomain window / buffer pointers enter provenance on allocate_domain and are revoked in _release_domain_now BEFORE the backend free (a commit barrier): the deferred window stays dispatchable, but once physical release begins the pointers are undispatchable, so a concurrent dispatch cannot validate a being-freed pointer and a partial backend-release failure leaves them dropped rather than "live forever". Boundary: catches stale-before-reuse, NOT strict ABA (a re-malloc of the same VA becomes live again) — that needs P1 generation handles. Python-only, no wire / C++ / ABI change; the raw-C++ deep-bypass (worker._orch._o.malloc) is out of scope. The remote-slot-ref capture runs after (not before) the kind4 provenance analysis, so an analysis failure cannot strand captured refs outside the rollback try and defer a remote free forever. Adds 45 device-free UT (provenance table incl. empty-entry fail-closed, target resolution, orchestrator memory ops incl. free commit-barrier + native-error + lock-held-across-native, single/group dispatch + resolved affinity + group length/duplicate-worker validation, capture-after-analysis ordering, CommDomain dispatch and release-before-free commit barrier, L2 path).
ChaoWao
added a commit
that referenced
this pull request
Jul 22, 2026
…rker (#1430) A child (kind4) device pointer is a bare int today — no owner, no generation (tensor.h child_memory is one bit). Once such a pointer entered a TaskArgs it could be freed, copied, or run on a worker that never allocated it: a raw device VA is not globally unique, so worker B silently touched worker A's (or illegal) device memory. Guard ① (#1397) only stopped host-addr rewrite from corrupting a device VA; nothing checked the pointer's owning worker. Track live child pointers by their exact (worker_id, ptr) provenance and validate every kind4 consumer (device ops and dispatch; L2 Worker.run is not a consumer): - Provenance table on the Worker: a typed (worker_id, ptr) -> entry map (malloc_owned + domain_allocation_ids), so a malloc base and an aliasing CommDomain buffer are distinct roles. Each op is atomic under one private lock; ordering is safety-first (record after alloc, revoke before free). Async interruption is fail-closed: a role is fully set before its entry is inserted, the last role is deleted directly (no role-less entry ever exists), and every live check tests the roles, not key presence, so an entry momentarily left empty never re-authorizes a freed pointer. Cleared on close(). - malloc: recorded after the backend malloc succeeds. free: a safety-first commit barrier — validate an exact live malloc base, then revoke provenance BEFORE the native free, so an async unwind (e.g. a KeyboardInterrupt after the binding returns) can never leave a freed address live; a native-free failure becomes a terminal leak, never a re-authorized maybe-freed address. copy_* require the device-side pointer live on that worker. Covers both Worker.malloc (L2 + L3) and the direct orch.malloc/copy/free paths (single Orchestrator choke), L2 and L3 unified. - submit_next_level / submit_next_level_group: a child_memory argument must resolve to exactly one eligible target worker (judged on the resolved eligibility, not a raw worker=-1) and be a live allocation there; 0 or >= 2 candidates is ambiguous and rejected. The resolved owner is passed to C++ as the effective affinity so the child TensorKey is keyed by its owner, not the raw -1 (otherwise the same buffer submitted once as -1 and once as W gets two keys and its dep is missed). The group path materialises a full per-member affinity only when workers is empty/None; a non-empty workers list is length- checked against args (never silently padded, which would bypass the C++ length check). A group must dispatch to distinct workers, so two members that resolve to the same owner (a duplicate non-negative affinity) are rejected rather than silently serialized on one WorkerThread. - CommDomain window / buffer pointers enter provenance on allocate_domain and are revoked in _release_domain_now BEFORE the backend free (a commit barrier): the deferred window stays dispatchable, but once physical release begins the pointers are undispatchable, so a concurrent dispatch cannot validate a being-freed pointer and a partial backend-release failure leaves them dropped rather than "live forever". Boundary: catches stale-before-reuse, NOT strict ABA (a re-malloc of the same VA becomes live again) — that needs P1 generation handles. Python-only, no wire / C++ / ABI change; the raw-C++ deep-bypass (worker._orch._o.malloc) is out of scope. The remote-slot-ref capture runs after (not before) the kind4 provenance analysis, so an analysis failure cannot strand captured refs outside the rollback try and defer a remote free forever. Adds 45 device-free UT (provenance table incl. empty-entry fail-closed, target resolution, orchestrator memory ops incl. free commit-barrier + native-error + lock-held-across-native, single/group dispatch + resolved affinity + group length/duplicate-worker validation, capture-after-analysis ordering, CommDomain dispatch and release-before-free commit barrier, L2 path).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make
Worker.init()the single, atomic startup point forlevel >= 3hierarchies.
init()now forks every local child, brings the whole subtree(recursively, for L4+) to READY, activates remote L3 sessions, starts the C++
scheduler, and only then commits READY — or fails after a bounded rollback that
leaves no child, mailbox, or session behind.
run()/create_host_buffer()/the remote register/memory APIs no longer implicitly trigger startup.
Builds on the readiness protocol from #1396.
Core lifecycle (
worker.py)NEW → INITIALIZING → READY|FAILEDcommit under one lock; noobservable started-but-not-initialized window.
_cleanup_partial_initis thesingle rollback entry.
init()is itself eager, so it publishesINIT_READYonly after its own subtree is ready.add_remote_l3_socket(health thread) movedafter the last local fork; sessions are only opened in
_init_hierarchical.ChipCallables and prewarm beforeINIT_READY,removing the unbounded post-READY
control_preparefrom the startup path(no C++ behavior change).
separate bounded grace covers cleanup.
_start_hierarchical()triggers removed.Cancellation domain (
worker.py)group. A mid-init child gets a cooperative
SIGTERMthat unwindsinner.init()and recursively reclaims its own grandchildren + nested shms;
killpgis thehard subtree backstop, so no orphan is left to the
resource_tracker.Remote path (
remote_l3_session.py,remote_l3_worker.py)snapshot; run the runner in its own session so the daemon reaps the whole
L3→L2subtree with a cooperative-SIGTERM-then-killpgcleanup.API linearization (
worker.py)register(local and remote),close,add_worker, andadd_remote_workerall linearize against an in-progress epoch instead of racing it.
SceneTest (
simpler_setup/scene_test.py)generate_argshost tensors aremoved into per-tensor born-shared
create_host_bufferstorage(dtype/shape/value preserved, non-contiguous rejected, LIFO release with
partial-construction rollback) so the eagerly-forked children can see them.
create_host_buffernow accepts a sub-only L3 (chip or sub child)._rewrite_blob_host_addrsskipschild_memory(device) tensors via a newbinding-exported offset, so a device pointer inside a host range is not
corrupted.
Testing
host-addr guard, startup readiness, subtree cancellation, API-race).
a2a3sim) L3 scene tests pass:multi_chip_dispatch,test_l3_group,test_l3_dependency,test_l3_host_buffer_registration,test_dynamic_register(5 cases).Two eager-only device-path issues were caught by the sim suite and fixed: the
chip
preparedset must carry from the pre-READY prepare into the dispatchloop, and two scene tests relied on a removed lazy "force-fork" trigger.