Add: investigation on the host worker dispatch latency budget - #1502
Conversation
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
After hw-native-sys#1499 took a single `Worker.run()` from 150 us to ~51 us, the obvious follow-up was whether another factor of three was available in what remained, and whether a blocking wakeup primitive (hw-native-sys#1498) was how to get it. Measured, and the answer to both is no. Recording it so the next person does not re-derive it. A bare fork+shm mailbox round trip with no simpler runtime in it costs 3.5 us against `Worker.run()`'s 53.8 us, which reads as "94% of dispatch latency is our code, not the IPC". That headline is an artifact of benchmarking a single task. Sweeping tasks per `run()` from 1 to 256 splits it into **43.3 us fixed per run() and 8.08 us marginal per task**: the cost is entering and leaving an orchestration, not dispatching work, and a production run submits a graph per `run()` rather than one task. At 256 tasks the fixed part is 2% of the invocation. The same numbers settle hw-native-sys#1498 against itself on latency grounds. A pipe wake measures 4.4 us one way on this box, so replacing the poll with a blocking wait adds more than the entire 3.5 us IPC floor to every dispatch, in exchange for CPU, while the latency it was meant to recover is not in the poll at all. What survives is narrower and unrelated: a child spinning while nothing is outstanding holds a core for as long as its parent lives — a CPU question for narrow-core hosts, already smaller since hw-native-sys#1495 made orphans exit with their parent. Also records the reconsideration triggers, since both conclusions are conditional: a workload that issues many small `run()` calls stops amortizing the fixed cost, and the CPU argument for a spin-then-block hybrid remains open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
9794ce1 to
4cf2cee
Compare
1 participant
Question
#1499 removed the parent's 50 us
TASK_DONEsleep and took a singleWorker.run()from 150 us to ~51 us. Two obvious follow-ups: is there another 3x in what remains, and is a blocking wakeup primitive (#1498) how to get it?Measured. Both answers are no. Recording it so the next person does not re-derive it.
The headline number is a trap
"94% of dispatch latency is our code, not the IPC" is the reading that invites a big optimisation push. It is an artifact of benchmarking a single task.
Sweeping tasks per
run():run()43.3 us fixed per
run(), 8.08 us marginal per task.The cost is entering and leaving an orchestration, not dispatching work — and a production run submits a graph per
run(), not one task. At 256 tasks the fixed part is 2% of the invocation.cProfileagrees on the shape: ~113 Python calls and ~15 threading lock/notify operations per no-op dispatch, spread thin with no hot frame to attack.The marginal 8.08 us against a 3.5 us bare round trip is ~2.3x, and that 2.3x buys arg marshalling, callable-identity resolution and per-task error reporting the bare pool does not do.
It also settles #1498 against itself
A pipe wake measures 4.4 us one way on this box (8.85 us round trip, p99 9.62 us). Replacing the poll with a blocking wait therefore adds more than the entire 3.5 us IPC floor to every dispatch, in exchange for CPU — while the latency it was supposed to recover is not in the poll at all.
What survives from #1498 is narrower and has nothing to do with latency: a child spinning while nothing is outstanding holds a core for as long as its parent lives. That is a CPU question for narrow-core hosts, and it got smaller when #1495 made orphans exit with their parent.
Conditional, and the entry says so
Both conclusions have reconsideration triggers recorded, because neither is unconditional:
run()calls (deep L3/L4 hierarchies, per-token orchestration) stops amortizing the 43 us. Measure its tasks-per-run()first — above ~16 the fixed cost is already under 25%.Per
.claude/rules/discipline.md§4 this lands indocs/investigations/with an index entry, since the outcome is "we measured it and did not do it".🤖 Generated with Claude Code