Repository navigation
fix(runner): bound a job's pids.max in tasks, and enforce the process limit by sampled process count - #18503
Merged
Conversation
… limit by sampled process count
chubes4
added a commit
that referenced
this pull request
Nov 18, 2026
…d resource-guard verdict (#18504) ## Problem A Lab gate job that ends before `gate-execute` prints anything was reported as a receipt parse error: > `validation.invalid_argument` … `Invalid argument 'gate.receipt': Lab gate gate_stdout unparseable (0 bytes) … EOF while parsing` That buried the real cause. In runner job `58a52bd1` (Cook `agent-task-a4a1fedf`), the runner's resource guard had stopped the gate with `resource_guard.process_count_limit_exceeded` and exit code 1. That verdict was already in the job's result event, under `metrics.guard_violation`. ## Change When the terminal gate stdout is empty, the transport returns a typed `runner.lab_transport_failure` before attempting to parse: > Lab gate job `<id>` ended with exit code N before printing its result: resource_guard.process_count_limit_exceeded: process_count=… (top: chrome-headless×79, …) - **Details:** `exit_code` and `resource_guard` (the violation), with `gate_disposition: unavailable`. - **No guard verdict:** without one, the message carries the redacted stderr tail instead. Private gates omit stderr. - **Retry behavior:** the error remains a retryable non-execution, so Cook defers the gate with the candidate preserved, as for other unavailable gates. The cause of this particular kill (`pids.max` set in the wrong unit) is fixed in #18503. ## Evidence - New `a_gate_that_printed_nothing_names_its_exit_and_resource_guard` covers both cases, guard verdict and stderr-only. It also checks the result is still classified as a Lab gate transport failure. - `-p homeboy-lab-runner --lib gate_transport` on the Lab: 21/21. --- Written by Claude (claude-opus via Claude Code): traced from the Lab runner job events and implemented this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#18441 wrote the runner's process-count guard limit straight into each job's
pids.max. The kernelpidscontroller counts tasks (threads), not processes, so the unit is 10–30× too strict for multithreaded workloads.On the Lab (limit 1024), the data-liberation-agent gate (
npm testwith headless Chrome) peaked at about 92 processes. That was 1024 tasks (pids.events max=9), so forks and thread creation failed, and the runner killed the job asresource_guard.process_count_limit_exceededwithprocess_count=1024. Runner job58a52bd1, Cookagent-task-a4a1fedf. The gate printed nothing, so the controller then reported a receipt parse error.Change
pids.maxisprocess limit × TASKS_PER_PROCESS (32)per job, andjobs/is that × concurrency. It stays a kernel fork-bomb backstop, now in the kernel's own unit.resource_guard_violation) overcgroup.procs, which counts processes. That was the unit before fix(runner): run every Lab job in its own kernel-enforced cgroup #18441.child_process_count_peakreport processes. The kernel message names tasks and the task limit separately.Evidence (Lab, delegated scope)
a_multithreaded_job_under_its_process_limit_runs: one Python process with 64 threads under an 8-process limit runs to completion, with no violation. It fails with the oldpids.max = limit, because Python cannot start its threads.-p homeboy-lab-runner --lib resource_metrics -- --test-threads=1gives 32/32.Written by Claude (claude-opus via Claude Code): traced the gate failure through the Lab runner job events and implemented this fix.