Skip to content

fix(runner): bound a job's pids.max in tasks, and enforce the process limit by sampled process count - #18503

Merged
chubes4 merged 1 commit into
mainfrom
fix/runner-job-pids-max-counts-tasks
Nov 18, 2026
Merged

chubes4 merged 1 commit into
mainfrom
fix/runner-job-pids-max-counts-tasks

Conversation

@chubes4

@chubes4 chubes4 commented Nov 18, 2026

Copy link
Copy Markdown
Member

Problem

#18441 wrote the runner's process-count guard limit straight into each job's pids.max. The kernel pids controller counts tasks (threads), not processes, so the unit is 10–30× too strict for multithreaded workloads.

On the Lab (limit 1024), the data-liberation-agent gate (npm test with headless Chrome) peaked at about 92 processes. That was 1024 tasks (pids.events max=9), so forks and thread creation failed, and the runner killed the job as resource_guard.process_count_limit_exceeded with process_count=1024. Runner job 58a52bd1, Cook agent-task-a4a1fedf. The gate printed nothing, so the controller then reported a receipt parse error.

Change

  • Kernel backstop in tasks. pids.max is process limit × TASKS_PER_PROCESS (32) per job, and jobs/ is that × concurrency. It stays a kernel fork-bomb backstop, now in the kernel's own unit.
  • Process limit in processes. The process limit is enforced by the existing sampled guard (resource_guard_violation) over cgroup.procs, which counts processes. That was the unit before fix(runner): run every Lab job in its own kernel-enforced cgroup #18441.
  • Reporting. The violation and child_process_count_peak report processes. The kernel message names tasks and the task limit separately.

Evidence (Lab, delegated scope)

  • New a_multithreaded_job_under_its_process_limit_runs: one Python process with 64 threads under an 8-process limit runs to completion, with no violation. It fails with the old pids.max = limit, because Python cannot start its threads.
  • The fork bomb, OOM group, cancel and daemon-leaf tests all pass: -p homeboy-lab-runner --lib resource_metrics -- --test-threads=1 gives 32/32.

Written by Claude (claude-opus via Claude Code): traced the gate failure through the Lab runner job events and implemented this fix.

@chubes4
chubes4 merged commit 39c3cec into main Nov 18, 2026
17 checks passed
@chubes4
chubes4 deleted the fix/runner-job-pids-max-counts-tasks branch November 18, 2026 22:19
chubes4 added a commit that referenced this pull request Nov 18, 2026
…d resource-guard verdict (#18504)

## Problem
A Lab gate job that ends before `gate-execute` prints anything was
reported as a receipt parse error:

> `validation.invalid_argument` … `Invalid argument 'gate.receipt': Lab
gate gate_stdout unparseable (0 bytes) … EOF while parsing`

That buried the real cause. In runner job `58a52bd1` (Cook
`agent-task-a4a1fedf`), the runner's resource guard had stopped the gate
with `resource_guard.process_count_limit_exceeded` and exit code 1. That
verdict was already in the job's result event, under
`metrics.guard_violation`.

## Change
When the terminal gate stdout is empty, the transport returns a typed
`runner.lab_transport_failure` before attempting to parse:

> Lab gate job `<id>` ended with exit code N before printing its result:
resource_guard.process_count_limit_exceeded: process_count=… (top:
chrome-headless×79, …)

- **Details:** `exit_code` and `resource_guard` (the violation), with
`gate_disposition: unavailable`.
- **No guard verdict:** without one, the message carries the redacted
stderr tail instead. Private gates omit stderr.
- **Retry behavior:** the error remains a retryable non-execution, so
Cook defers the gate with the candidate preserved, as for other
unavailable gates.

The cause of this particular kill (`pids.max` set in the wrong unit) is
fixed in #18503.

## Evidence
- New `a_gate_that_printed_nothing_names_its_exit_and_resource_guard`
covers both cases, guard verdict and stderr-only. It also checks the
result is still classified as a Lab gate transport failure.
- `-p homeboy-lab-runner --lib gate_transport` on the Lab: 21/21.

---
Written by Claude (claude-opus via Claude Code): traced from the Lab
runner job events and implemented this change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant