Fix: bound Remote L3 startup by one root budget, not per-remote timeout - #1420
Merged
ChaoWao merged 1 commit intoJul 21, 2026
Merged
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughRemote L3 startup now propagates a finite remaining startup budget through manifests, hierarchical activation, socket handshakes, endpoint attachment, and nested worker initialization. Validation and tests cover fallback behavior, deadline consumption, deferred activation, and expired budgets. ChangesRemote startup budget
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Worker
participant RemoteL3Daemon
participant EndpointRegistry
Worker->>RemoteL3Daemon: open session with remaining startup budget
RemoteL3Daemon-->>Worker: report ready and return socket
Worker->>EndpointRegistry: register remote socket with bounded attach timeout
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a single-root startup budget mechanism for remote L3 sessions, ensuring that all remote sessions draw from a single shared root deadline rather than multiplying the startup budget. The remaining budget is tracked and propagated via the startup_remaining_s manifest field. Additionally, session opening and activation are deferred from _init_hierarchical (pre-fork) to _activate_remote_sessions (post-fork) to ensure they occur after the last local fork. Timeout values are also validated to be positive and finite. Comprehensive unit tests have been added to verify these changes. I have no feedback to provide as there are no review comments.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/simpler/remote_l3_worker.py`:
- Around line 76-83: Fix _startup_remaining_s in both
python/simpler/remote_l3_worker.py lines 76-83 and
python/simpler/remote_l3_session.py lines 172-180 to call
_session_timeout_s(manifest) only when "startup_remaining_s" is absent;
otherwise validate the provided value directly. Add a TestRemoteSideValidation
case in tests/ut/py/test_worker/test_remote_startup_budget.py covering valid
startup_remaining_s with invalid session_timeout_s.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: afc37af4-2281-4a6c-b73a-19e4a6706b35
📒 Files selected for processing (6)
python/simpler/remote_l3_session.pypython/simpler/remote_l3_worker.pypython/simpler/worker.pytests/ut/py/test_callable_identity.pytests/ut/py/test_remote_l3_lifecycle.pytests/ut/py/test_worker/test_remote_startup_budget.py
YunjiQin
force-pushed
the
worktree-p0.3-sim-remote-correctness
branch
4 times, most recently
from
July 21, 2026 09:39
aecb4bd to
a1a16ef
Compare
P0.3 simulation remote correctness. Adding Remote L3 workers previously multiplied the startup budget and mixed the runtime command timeout into the startup path. Fix the root-owned single-deadline contract. - Propagate a distinct manifest field startup_remaining_s: root keeps one absolute startup deadline and, before each _open_remote_session, grants the remaining slice as both the daemon socket timeout and the manifest startup_remaining_s. The remote rebuilds its own deadline against this host's own monotonic clock (no cross-host absolute timestamp) and bounds its L3->L2 subtree by that slice instead of a fresh full session_timeout_s. - Keep startup_remaining_s separate from remote_session_timeout_s, which still bounds the runtime command socket; the two must not be conflated. - Defer opening AND activating remote sessions into _activate_remote_sessions, called only after this process's last local fork, so opening (which starts the remote subtree) and endpoint registration (which spawns the health thread) both stay behind every local fork. - Validate startup/session timeouts as positive-finite on parent, session, and daemon (untrusted-wire) paths before creating any resource. The session_timeout_s fallback for a pre-P0.3 parent that omits startup_remaining_s is evaluated only when the key is absent, so a valid startup_remaining_s is never rejected because of an unrelated invalid session_timeout_s. - Cover budget non-multiplication, deferred activation, failure propagation, and positive-finite validation with device-free UTs.
ChaoWao
approved these changes
Jul 21, 2026
3 tasks
2 tasks
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…runtime timeout hw-native-sys#1420 delivered the shape of a single-root remote-L3 startup budget (a duration on the wire, a remote-local monotonic clock, activation after the last fork) but left two defects on the seam where a remote's timeout is derived and passed on. R2 [high] — the root deadline was not enforced end-to-end. The parent computed a remaining startup duration once, then reused it as a fresh full relative timeout at every blocking stage (connect / send / framed recv, and the native command-connect / HELLO / health-connect), and never re-checked the deadline after attach before committing READY. A slow peer or several stacked stages could exceed the root wall-clock budget. R4 [medium] — the runtime command timeout was polluted by leftover startup budget. add_remote_l3_socket received a single min(session_timeout, remaining); RemoteL3SocketTransport reused it as the per-runtime-command deadline, so a remote attached late in startup got its runtime timeout clamped to that leftover value and legitimate long commands timed out as ENDPOINT_FAILURE. Both are fixed by deriving every blocking point's remaining from one host-local absolute deadline and fully separating startup budget from the runtime command timeout. Duration stays on the wire; no manifest field changes. Python (worker.py): - _open_remote_session takes an absolute deadline. Name resolution and every per-address connect go through _connect_within_deadline / _resolve_within_deadline, which bound getaddrinfo (in a daemon thread) and each connect by the deadline — mirroring the C++ connect_tcp_socket — instead of socket.create_connection, which grants a fresh full timeout to every address and never bounds DNS. The recv path re-derives its timeout before each recv. A _remaining_until helper raises TimeoutError on a spent slice instead of settimeout(0.0) (which would flip the socket to non-blocking and leak BlockingIOError). The wire startup_remaining_s and the send socket timeout are derived after the manifest is built, right before send, so they reflect what is actually left of the budget. - _activate_remote_sessions threads the deadline through, passes attach and runtime timeouts separately, and rechecks the deadline after the last attach. _start_hierarchical rechecks once more after local endpoint registration and scheduler start, immediately before init() publishes READY, so post-attach work cannot commit READY past the root deadline. C++ (remote_endpoint.{h,cpp}, worker.{h,cpp}, worker_bind.h): - RemoteL3SocketTransport takes attach_timeout_s + runtime_timeout_s and builds one attach_deadline_ before connect_socket(). Command-connect, the HELLO read, and health-connect all derive from attach_deadline_; runtime commands and the health-monitor loop use runtime_timeout_s_. wait_readable/wait_writable/read_frame/write_all are parameterized with an absolute deadline. add_remote_l3_socket and its binding gain both timeouts (internal C++ ABI; not wire). Tests are repro-first (discipline §3): - Python: a deterministic _FakeClock; the single-root-budget suite is rewritten to the deadline API (deadline identity across remotes + a shrinking derived slice), plus attach-vs-runtime split, a final-recheck test, per-op fail-fast / send-time-recompute tests, bounded-connect tests (resolution and each address bounded by the shrinking remaining, fail-fast past the deadline), and a wire-stays-duration guard. The behavioral tests were confirmed to fail against the pre-fix worker.py. - cpput: the ctor construction updates to six args (compile break vs the old signature); new tests prove the HELLO read is bounded by the attach timeout and that a runtime read survives an already-elapsed attach deadline (the definitive value-split proof), plus ctor validation of both timeouts.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…runtime timeout hw-native-sys#1420 delivered the shape of a single-root remote-L3 startup budget (a duration on the wire, a remote-local monotonic clock, activation after the last fork) but left two defects on the seam where a remote's timeout is derived and passed on. R2 [high] — the root deadline was not enforced end-to-end. Every blocking stage rebuilt a fresh full relative timeout instead of deriving from a single deadline, and there was no deadline recheck before READY. R4 [medium] — the runtime command timeout was clamped to the leftover startup budget (min(session_timeout, remaining) reused as the per-command deadline), so a late-attached remote failed long runtime commands as ENDPOINT_FAILURE. Both are fixed by deriving every blocking point's remaining from one host-local absolute deadline and fully separating startup budget from the runtime command timeout. Duration stays on the wire; no manifest field changes. Parent (worker.py): - _open_remote_session(deadline): name resolution and every per-address connect go through _connect_within_deadline / _resolve_within_deadline and derive their remaining from the deadline (resolution is synchronous — no abandoned getaddrinfo thread that could deadlock a later fork), and the recv path re-derives its timeout per recv. _remaining_until raises TimeoutError on a spent slice instead of settimeout(0.0). The wire startup_remaining_s and the send timeout are derived after the manifest build, right before send. - _activate_remote_sessions passes attach and runtime timeouts separately and rechecks after the last attach. The final deadline gate lives in the same READY-commit critical section as the lifecycle publish, so a thread descheduled after attach cannot commit READY late. Remote (remote_l3_worker.py, remote_l3_session.py): - The daemon builds one absolute deadline up front; the runner-ready wait derives from it (its own Popen/setup time is charged, not restarted), and it hands the runner the current remaining, not the original slice. - The runner establishes its single deadline before registry install, so registry + inner init draw from it rather than a fresh slice. The whole parent -> daemon -> runner chain now enforces one budget instead of resetting it at each hop. C++ (remote_endpoint.{h,cpp}, worker.{h,cpp}, worker_bind.h): - RemoteL3SocketTransport takes attach_timeout_s + runtime_timeout_s and builds one attach_deadline_ before connect_socket(). Command-connect, the HELLO read, and health-connect all derive from attach_deadline_; runtime commands and the health-monitor loop use runtime_timeout_s_. wait_readable/wait_writable/read_frame/write_all are parameterized with an absolute deadline. add_remote_l3_socket and its binding gain both timeouts (internal C++ ABI; not wire). Tests are repro-first (discipline §3): a deterministic fake clock; the single-root-budget suite rewritten to the deadline API; attach-vs-runtime split; final-recheck and READY-commit-gate tests; per-op fail-fast / send-time recompute; bounded-connect (resolution and each address bounded by the shrinking remaining); daemon-hands-runner-decremented-budget and runner-builds-deadline-before-registry regressions; and cpput tests proving the HELLO read is bounded by the attach timeout and a runtime read survives an already-elapsed attach deadline. Behavioral tests were confirmed to fail against the pre-fix sources.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…plit attach vs runtime timeout hw-native-sys#1420 delivered the shape of a single-root remote-L3 startup budget but left the budget resettable at several hops, and clamped the runtime command timeout to leftover startup budget. R2 [high] — make the single root deadline a hard wall across the whole parent -> daemon -> runner -> native-attach chain. R4 [medium] — separate the attach timeout from the post-READY runtime command timeout. Duration stays on the cross-machine wire; no cross-machine wire fields. Parent (worker.py): - _open_remote_session(deadline): resolution is numeric-only via AI_NUMERICHOST (getaddrinfo can hang uncancellably; a hostname is rejected outright rather than pinning init() in INITIALIZING); every per-address connect and each recv derive from the deadline; _remaining_until raises TimeoutError on a spent slice; the wire budget and send timeout are derived after the manifest build. - The final root-deadline gate lives in the same READY-commit critical section and applies to every hierarchical worker (a local child may have remote descendants), so a thread descheduled after startup cannot publish READY late. Remote (remote_l3_worker.py, remote_l3_session.py): - The daemon builds one absolute deadline up front and hands the runner that ABSOLUTE monotonic deadline (same host, shared CLOCK_MONOTONIC). The runner uses it verbatim, so its deadline equals the daemon's and spawn/import/registry time is charged instead of restarting a frozen duration. The runner-ready wait derives from the same deadline; the runner establishes it before registry install. C++ (remote_endpoint.{h,cpp}, worker.{h,cpp}, worker_bind.h): - RemoteL3SocketTransport takes attach_timeout_s + runtime_timeout_s and one attach_deadline_. connect_tcp_socket now takes the absolute Deadline directly (resolution + each connect derive from it — no now()+remaining re-derivation that a descheduled thread could stretch). Command/HELLO/ health connect use attach_deadline_; runtime frame I/O + the health loop use runtime_timeout_s_. Runtime writes use MSG_DONTWAIT so a stalled reader cannot blow the deadline in one blocking send() of a large frame; write_all re-polls under the deadline on EAGAIN. add_remote_l3_socket and its binding gain both timeouts (internal C++ ABI; not wire). Tests are repro-first (discipline §3): a deterministic fake clock; the single-root-budget suite on the deadline API; attach-vs-runtime split; READY-commit-gate (unconditional for hierarchical startup); per-op fail-fast / send-time recompute; bounded-connect with numeric-only resolution (hostname rejected, AI_NUMERICHOST asserted); daemon-hands-runner-absolute-deadline and runner-uses-it-verbatim regressions; and cpput tests proving the HELLO read is bounded by the attach timeout, a runtime read survives an elapsed attach deadline, and a runtime write to a stalled reader times out at the runtime deadline. Behavioral tests were confirmed to fail against the pre-fix sources.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…plit attach vs runtime timeout hw-native-sys#1420 delivered the shape of a single-root remote-L3 startup budget but left the budget resettable at several hops, and clamped the runtime command timeout to leftover startup budget. R2 [high] — make the single root deadline a hard wall across the whole parent -> daemon -> runner -> native-attach chain. R4 [medium] — separate the attach timeout from the post-READY runtime command timeout. The cross-machine parent -> daemon wire carries only a duration; no cross-machine wire fields are added. Parent (worker.py): - Remote endpoints are numeric-only (or localhost), validated in add_remote_worker before any startup resource: getaddrinfo resolution is unbounded and uncancellable, so a hostname is rejected up front rather than mid-activation (which would roll back the whole forked tree). RemoteWorkerSpec and the design doc state the contract. - _open_remote_session(deadline): AI_NUMERICHOST resolution cannot block; every per-address connect and each recv derive from the deadline; the wire budget and send timeout are derived after the manifest build. - The final root-deadline gate lives in the READY-commit critical section and applies to every hierarchical worker (a local child may have remote descendants), so a thread descheduled after startup cannot publish READY late. Remote (remote_l3_worker.py, remote_l3_session.py): - The daemon builds one absolute deadline up front and, over the private same-host daemon -> runner manifest, hands the runner its ABSOLUTE monotonic deadline (shared CLOCK_MONOTONIC). The runner uses it verbatim, so its deadline equals the daemon's and spawn/import/registry time is charged; it establishes the deadline before registry install. - Waiting for the parent to attach is still the attach phase: command accept is bounded by the remaining startup budget, not session_timeout_s. The post-attach command lane uses session_timeout_s (R4). C++ (remote_endpoint.{h,cpp}, worker.{h,cpp}, worker_bind.h): - RemoteL3SocketTransport takes attach_timeout_s + runtime_timeout_s and one attach_deadline_. connect_tcp_socket takes the absolute Deadline directly (resolution + each connect derive from it). Command/HELLO/ health connect use attach_deadline_; runtime frame I/O + the health loop use runtime_timeout_s_. The connected fd stays O_NONBLOCK for its whole life and all frame I/O polls for readiness under a deadline, so a large write to a stalled reader cannot outlast the deadline in one blocking send() (the earlier per-send MSG_DONTWAIT was not portable and hung ~47s on macOS); recv/read paths handle EAGAIN. add_remote_l3_socket and its binding gain both timeouts (internal C++ ABI; not wire). Tests are repro-first (discipline §3): deterministic fake clock; the single-root-budget suite on the deadline API; attach-vs-runtime split; READY-commit-gate (unconditional for hierarchical startup); numeric-only endpoint contract (hostname rejected at registration); command-accept bounded by the startup deadline (distinct from session_timeout); daemon-hands-runner-absolute-deadline and runner-uses-it-verbatim regressions; and cpput tests proving the HELLO read is bounded by the attach timeout, a runtime read survives an elapsed attach deadline, and a runtime write to a stalled reader times out at the runtime deadline.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…plit attach vs runtime timeout hw-native-sys#1420 delivered the shape of a single-root remote-L3 startup budget but left the budget resettable at several hops, and clamped the runtime command timeout to leftover startup budget. R2 [high] — make the single root deadline a hard wall across the whole parent -> daemon -> runner -> native-attach chain. R4 [medium] — separate the attach timeout from the post-READY runtime command timeout. The cross-machine parent -> daemon wire carries only a duration; no cross-machine wire fields are added. Parent (worker.py): - Remote endpoints are numeric-only (or localhost), validated in add_remote_worker before any startup resource: getaddrinfo resolution is unbounded and uncancellable, so a hostname is rejected up front rather than mid-activation (which would roll back the whole forked tree). RemoteWorkerSpec and the design doc state the contract. - _open_remote_session(deadline): AI_NUMERICHOST resolution cannot block; every per-address connect and each recv derive from the deadline; the wire budget and send timeout are derived after the manifest build. - The final root-deadline gate lives in the READY-commit critical section and applies to every hierarchical worker (a local child may have remote descendants), so a thread descheduled after startup cannot publish READY late. Remote (remote_l3_worker.py, remote_l3_session.py): - The daemon builds one absolute deadline up front and, over the private same-host daemon -> runner manifest, hands the runner its ABSOLUTE monotonic deadline (shared CLOCK_MONOTONIC). The runner uses it verbatim, so its deadline equals the daemon's and spawn/import/registry time is charged; it establishes the deadline before registry install. - Waiting for the parent to attach is still the attach phase: command accept is bounded by the remaining startup budget, not session_timeout_s. The command connection then stays blocking: the command loop idle-waits for the next frame, so a finite timeout would tear down a healthy idle session; parent death is caught by the health lane. C++ (remote_endpoint.{h,cpp}, worker.{h,cpp}, worker_bind.h): - RemoteL3SocketTransport takes attach_timeout_s + runtime_timeout_s and one attach_deadline_. connect_tcp_socket takes the absolute Deadline directly (resolution + each connect derive from it). Command/HELLO/ health connect use attach_deadline_; runtime frame I/O + the health loop use runtime_timeout_s_. The connected fd stays O_NONBLOCK for its whole life and all frame I/O polls for readiness under a deadline, so a large write to a stalled reader cannot outlast the deadline in one blocking send() (the earlier per-send MSG_DONTWAIT was not portable and hung ~47s on macOS); recv/read paths handle EAGAIN. add_remote_l3_socket and its binding gain both timeouts (internal C++ ABI; not wire). Tests are repro-first (discipline §3): deterministic fake clock; the single-root-budget suite on the deadline API; attach-vs-runtime split; READY-commit-gate (unconditional for hierarchical startup); numeric-only endpoint contract (hostname rejected at registration); command-accept bounded by the startup deadline (distinct from session_timeout); daemon-hands-runner-absolute-deadline and runner-uses-it-verbatim regressions; and cpput tests proving the HELLO read is bounded by the attach timeout, a runtime read survives an elapsed attach deadline, and a runtime write to a stalled reader times out at the runtime deadline.
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Jul 22, 2026
…plit attach vs runtime timeout hw-native-sys#1420 delivered the shape of a single-root remote-L3 startup budget but left the budget resettable at several hops, and clamped the runtime command timeout to leftover startup budget. R2 [high] — make the single root deadline a hard wall across the whole parent -> daemon -> runner -> native-attach chain. R4 [medium] — separate the attach timeout from the post-READY runtime command timeout. The cross-machine parent -> daemon wire carries only a duration; no cross-machine wire fields are added. Parent (worker.py): - Remote endpoints are numeric-only (or localhost), validated in add_remote_worker before any startup resource: getaddrinfo resolution is unbounded and uncancellable, so a hostname is rejected up front rather than mid-activation (which would roll back the whole forked tree). RemoteWorkerSpec and the design doc state the contract. - _open_remote_session(deadline): AI_NUMERICHOST resolution cannot block; every per-address connect and each recv derive from the deadline; the wire budget and send timeout are derived after the manifest build. - The final root-deadline gate lives in the READY-commit critical section and applies to every hierarchical worker (a local child may have remote descendants), so a thread descheduled after startup cannot publish READY late. Remote (remote_l3_worker.py, remote_l3_session.py): - The daemon builds one absolute deadline up front and, over the private same-host daemon -> runner manifest, hands the runner its ABSOLUTE monotonic deadline (shared CLOCK_MONOTONIC). The runner uses it verbatim, so its deadline equals the daemon's and spawn/import/registry time is charged; it establishes the deadline before registry install. - Waiting for the parent to attach is still the attach phase: command accept is bounded by the remaining startup budget, not session_timeout_s. The command connection then stays blocking: the command loop idle-waits for the next frame (forced blocking, since accept() inherits the process default timeout), so a finite timeout would tear down a healthy idle session; the loop ends when the parent closes the command socket (EOF). C++ (remote_endpoint.{h,cpp}, worker.{h,cpp}, worker_bind.h): - RemoteL3SocketTransport takes attach_timeout_s + runtime_timeout_s and one attach_deadline_. connect_tcp_socket takes the absolute Deadline directly (resolution + each connect derive from it). Command/HELLO/ health connect use attach_deadline_; runtime frame I/O + the health loop use runtime_timeout_s_. The connected fd stays O_NONBLOCK for its whole life and all frame I/O polls for readiness under a deadline, so a large write to a stalled reader cannot outlast the deadline in one blocking send() (the earlier per-send MSG_DONTWAIT was not portable and hung ~47s on macOS); recv/read paths handle EAGAIN. add_remote_l3_socket and its binding gain both timeouts (internal C++ ABI; not wire). Tests are repro-first (discipline §3): deterministic fake clock; the single-root-budget suite on the deadline API; attach-vs-runtime split; READY-commit-gate (unconditional for hierarchical startup); numeric-only endpoint contract (hostname rejected at registration); command-accept bounded by the startup deadline (distinct from session_timeout); daemon-hands-runner-absolute-deadline and runner-uses-it-verbatim regressions; and cpput tests proving the HELLO read is bounded by the attach timeout, a runtime read survives an elapsed attach deadline, and a runtime write to a stalled reader times out at the runtime deadline.
ChaoWao
added a commit
that referenced
this pull request
Jul 22, 2026
…plit attach vs runtime timeout (#1432) #1420 delivered the shape of a single-root remote-L3 startup budget but left the budget resettable at several hops, and clamped the runtime command timeout to leftover startup budget. R2 [high] — make the single root deadline a hard wall across the whole parent -> daemon -> runner -> native-attach chain. R4 [medium] — separate the attach timeout from the post-READY runtime command timeout. The cross-machine parent -> daemon wire carries only a duration; no cross-machine wire fields are added. Parent (worker.py): - Remote endpoints are numeric-only (or localhost), validated in add_remote_worker before any startup resource: getaddrinfo resolution is unbounded and uncancellable, so a hostname is rejected up front rather than mid-activation (which would roll back the whole forked tree). RemoteWorkerSpec and the design doc state the contract. - _open_remote_session(deadline): AI_NUMERICHOST resolution cannot block; every per-address connect and each recv derive from the deadline; the wire budget and send timeout are derived after the manifest build. - The final root-deadline gate lives in the READY-commit critical section and applies to every hierarchical worker (a local child may have remote descendants), so a thread descheduled after startup cannot publish READY late. Remote (remote_l3_worker.py, remote_l3_session.py): - The daemon builds one absolute deadline up front and, over the private same-host daemon -> runner manifest, hands the runner its ABSOLUTE monotonic deadline (shared CLOCK_MONOTONIC). The runner uses it verbatim, so its deadline equals the daemon's and spawn/import/registry time is charged; it establishes the deadline before registry install. - Waiting for the parent to attach is still the attach phase: command accept is bounded by the remaining startup budget, not session_timeout_s. The command connection then stays blocking: the command loop idle-waits for the next frame (forced blocking, since accept() inherits the process default timeout), so a finite timeout would tear down a healthy idle session; the loop ends when the parent closes the command socket (EOF). C++ (remote_endpoint.{h,cpp}, worker.{h,cpp}, worker_bind.h): - RemoteL3SocketTransport takes attach_timeout_s + runtime_timeout_s and one attach_deadline_. connect_tcp_socket takes the absolute Deadline directly (resolution + each connect derive from it). Command/HELLO/ health connect use attach_deadline_; runtime frame I/O + the health loop use runtime_timeout_s_. The connected fd stays O_NONBLOCK for its whole life and all frame I/O polls for readiness under a deadline, so a large write to a stalled reader cannot outlast the deadline in one blocking send() (the earlier per-send MSG_DONTWAIT was not portable and hung ~47s on macOS); recv/read paths handle EAGAIN. add_remote_l3_socket and its binding gain both timeouts (internal C++ ABI; not wire). Tests are repro-first (discipline §3): deterministic fake clock; the single-root-budget suite on the deadline API; attach-vs-runtime split; READY-commit-gate (unconditional for hierarchical startup); numeric-only endpoint contract (hostname rejected at registration); command-accept bounded by the startup deadline (distinct from session_timeout); daemon-hands-runner-absolute-deadline and runner-uses-it-verbatim regressions; and cpput tests proving the HELLO read is bounded by the attach timeout, a runtime read survives an elapsed attach deadline, and a runtime write to a stalled reader times out at the runtime deadline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
P0.3 — simulation Remote L3 startup budget / activation correctness. Adding
Remote L3 workers previously multiplied the startup budget (each remote got
a fresh full
session_timeout_s) and conflated the runtime command timeout withthe startup path. This lands the root-owned single-deadline contract from the L3
roadmap.
deadline; before each
_open_remote_session()it computes the remaining sliceand grants it as both the daemon socket timeout and a new, distinct manifest
field
startup_remaining_s. The remote rebuilds its own deadline against itslocal monotonic clock (no cross-host absolute timestamp) and bounds its
L3→L2 subtree by that slice instead of a fresh
session_timeout_s.startup_remaining_sis kept separate fromremote_session_timeout_s.The latter still bounds the runtime command socket; mixing them would freeze
remaining startup budget into a runtime command deadline.
starts the remote subtree on its own clock) and endpoint registration (which
spawns the
RemoteL3Endpointhealth thread) both move into_activate_remote_sessions, called only after this process's last local fork,preserving the fork-before-thread invariant.
startup/session timeouts now fail on parent, session, and daemon (untrusted
wire) paths before creating resources. A pre-P0.3 parent that omits
startup_remaining_sfalls back to the session timeout.Exit criteria covered
Not in scope (tracked elsewhere): CLEANUP_ACK, lease/epoch, partition recovery.
Testing
test_remote_startup_budget.py,test_remote_l3_lifecycle.py,test_callable_identity.py(115 passed).New UTs cover budget non-multiplication (shrinking per-remote slice),
deferred activation, fail-fast on expired deadline, and positive-finite
validation on both parent and untrusted-wire paths.