Infra: verifier/env reliability fixes + offline reshard override#40
Merged
Conversation
… apt-get shim Three container-environment fixes for the separate verifier, grouped: - share a built verifier context across runs of the same task to avoid rebuilding it every trial - route the separate-verifier container through the apt/pip caches with the same retry/timeout policy the agent container already gets, so a flaky mirror cannot stall the grade - restore the real apt-get on images that ship a hermetic no-op `apt-get` shim (keeping the real binary at apt-get.real), so offline verifies can install packages the image did not pre-bake (e.g. git-multibranch's `expect`). No-op on images without the shim.
OrbStack injects a no_proxy/NO_PROXY into every container that lists its internal IPv6 ULA range (e.g. ::1, fd07:...::/64). httpx cannot parse a bare IPv6 entry in no_proxy -- it splits on ':' and reads the address bytes as a port, raising InvalidURL at Client construction -- so any task that builds an httpx client (notably huggingface_hub v1.x via datasets) crashes before its first request and silently falls back to the offline cache. The injection is regenerated on every OrbStack restart, so a host-side edit does not stick. Probe the container's injected no_proxy once and pin a cleaned override (IPv6 entries dropped, everything else preserved) into the exec env so every agent action and the in-container verifier inherit it. Best-effort and a no-op on hosts that inject no IPv6. Refs: encode/httpx#3741, psf/requests#6313, orbstack/orbstack#2449
A baseline run killed mid-flight leaves state.json pointing the active baseline at a record that never finalized as a keep. _ensure_baseline_at_head trusted that pointer on commit+panels alone, so on restart it short-circuited (returned False) and let the loop compare candidates against partial, ungraded evidence instead of measuring a fresh baseline. Discard a non-keep/unconcluded active baseline before the short-circuit: finalize the stale record as a crash and fall back to its parent keep (or seed fresh when there is none), so an interrupted baseline reliably restarts fresh. Adds a regression test reproducing the short-circuit.
Annotate shared_verifier_task_names with the concrete reason each task needs to reuse the agent's live container instead of a fresh separate verifier (running daemon the verifier connects to / build artifacts or installed deps the agent produced / git over the agent's working tree / local-only image), and add the tasks that fail for those same reasons: git-multibranch, mailman, adaptive-rejection-sampler, git-leak-recovery, merge-diff-arc-agi-task, sanitize-git-repo, reshard-c4-data.
The previous 600s ceiling truncated long-running build/verify steps. Raise the default per-turn timeout so slow tasks are bounded by the trial budget rather than cut off mid-action.
Drop reshard-c4-data from the train panel and add it to a contamination_risk excluded group instead of carrying a local task override for it. Running the task offline (to avoid re-downloading the held-out HF c4 shard every trial) would require an override that commits the canonical solution and verifier oracle into this repo, which the self-improvement loop could read. Excluding it removes that risk; the task also exceeds the train task_timeout_sec.
workofart
force-pushed
the
infra-verifier-env-fixes
branch
from
10cb5c1 to
e593dce
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Collects the trustworthy infra work off
exp-0601onto a clean branch offmain, dropping the auto-generatedcandidate exp-*commits and the unrelated concurrency bump, and adds two new container-environment fixes. Goal: a stable harness souv run autoresults can be trusted.Commits
apt-geton images shipping a hermetic no-op shim (real binary kept atapt-get.real) so offline verifies can install packages the image didn't pre-bake (e.g. git-multibranch'sexpect). No-op on images without the shim.no_proxylisting its internal IPv6 ULA range (::1,fd07:...::/64); httpx can't parse a bare IPv6 entry and raisesInvalidURLatClient()construction, so any task using httpx (e.g.huggingface_hubv1 viadatasets) crashes and silently falls back to the offline cache. OrbStack rewrites~/.docker/config.jsonon every restart, so a host edit doesn't stick — instead we probe the container's injectedno_proxyand pin a cleaned override into the exec env. Best-effort; no-op without IPv6. Refs Fix no_proxy to support IPV6 CIDR format encode/httpx#3741, requests ignores no_proxy which contains a IPv6 CIDR psf/requests#6313, Orbstack 2.1.2 🔥 bug: HTTP_PROXY env vars injected into containers despitenetwork_proxy: none, breaking any inter-container HTTP(S) on user-defined Docker networks orbstack/orbstack#2449.contamination_riskexcluded group instead of carrying a local task override. Running it offline (to avoid re-downloading the held-out HF c4 shard every trial) would require an override that commits the canonical solution and verifier oracle into this repo, which the self-improvement loop could read; excluding it removes that risk (the task also exceeds the traintask_timeout_sec).Verification
InvalidURL: Invalid port: 'b51a:cc66:f0::') on OrbStack's actual injectedno_proxy, and confirmed the stripped value letshttpx.Client()construct while preserving every*.orb.internal/host.docker.internal/ IPv4 entry.HarnessConfigvalidates (panels/excluded disjoint);tests/adapters/test_env.py+tests/control/test_supervisor.py+tests/harness/— 167 passed; ruff clean.