agent: coop prompt told agents the shared git remote was read-only - #80
Merged
Conversation
The mini_swe_agent_v2 coop prompt titled its git section "Shared Git Remote (read-only)", listed only fetch/log/diff under "Allowed (read-only)", never mentioned push, and instructed "Do not merge, pull, or rebase their branch into yours". Both claims are wrong. GitConnector runs `git daemon --enable=receive-pack`, so the remote accepts writes, and grading never reads those branches: the evaluator applies each agent's submitted patch.txt to branches it creates itself (eval/sandbox.py). Agents are in separate containers, so a push is the only way code can cross between them. Measured across 117 trajectories in three flash_10 runs: 292 `git fetch team` and 0 `git push team`. Every fetch returned the untouched baseline, because 0.0.22 only publishes a patch at exit -- by which point the peer can no longer act on it. Agents were following their instructions exactly. This prompt was the only surface that said so. GitConnector's docstring already advertised `git push team <branch>`, and agents/_coop/prompt.py (used by the claude_code and codex adapters) already tells agents to "push so peers can fetch you". The section now describes a read/write remote and tells agents to commit and push as they work. The blanket prohibition is replaced by the one constraint that actually affects grading: the submitted patch.txt must contain only that agent's own changes, since the two patches are merged and duplicate edits break the merge. Local fetch/merge is explicitly allowed -- the rule is about what you submit, not what you do in your worktree -- with `git diff -- <paths>` given as the way to scope a patch when both agents touched the same file. The rewritten section is shorter than the one it replaces (1210 vs 1713 chars) and survives context compaction, which preserves the task message verbatim.
Submission was a unified diff the agent wrote to a local file. Nothing else could see it, so the artifact that got graded was decoupled from anything a colleague could read, and an agent could submit work it had never shared. The agent now commits what it wants to submit, pushes its branch and opens a PR; the PR is what gets graded. A `gh` shim implements gh pr create/list/view/ diff/checkout over the shared remote using plain git, so agents use a spelling they already know rather than a command invented for this benchmark. A PR tracks its branch, as on a forge: agents are told to open one early so a colleague has context, and later commits are included once pushed. An agent that never opens a PR submits nothing and scores zero. That is an agent failure, and it is logged so it stays attributable rather than looking like failed tests. Solo uses the same path against a bare repo in its own sandbox, so there is one submission mechanism rather than two that can drift apart -- extraction had already moved to the PR while solo.yaml still told agents to write patch.txt, which silently discarded every solo submission. Both prompts got smaller: patch.txt needed ~30 lines teaching a diff incantation that exists nowhere in real engineering; gh pr create needs none. Also fixed, all found by reviewing this change against a live run: - Coop agents could read the whole upstream history, including commits made AFTER the task commit. A task image clones then checks out a sha, which leaves the future reachable through refs/remotes, tags, and the local branch the clone left at the tip -- checkout only detaches HEAD. For a task derived from a real PR, `git log --all -p` could show the upstream implementation of the feature being asked for. Setup now removes the remote, deletes those refs, expires the reflog and prunes, so the objects are gone rather than merely unreferenced. - A submission could be re-baselined by anyone pushing to main: patches were diffed against a movable ref on a daemon with no access control. The base commit is pinned at setup. - The modal backend decoded sandbox output as strict UTF-8, so an agent that read a binary file (observed: `tail .git/index`) killed its own run with a decode error recorded as an agent failure. It now decodes leniently, as the docker backend already did. - A failed `gh` shim install was a log warning; without the shim the agent cannot submit at all. Setup now raises. - The shared remote is `origin` and the clone's unreachable upstream is removed, so the remote agents reach for by reflex is the one that works. Verified on real infrastructure (flask_task/5526): both agents negotiated a shared signature over messaging, agent1 read the peer's branch with git diff, implemented both parameters, opened a PR; 120 lines extracted, merged clean, both feature suites passed. The same pair failed before these changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
mini_swe_agent_v2's coop prompt titled its git section "Shared Git Remote (read-only)", listed onlyfetch/log/diffunder "Allowed (read-only)", never mentionedpush, and instructed "Do not merge, pull, or rebase their branch into yours".Both claims are wrong:
GitConnectorrunsgit daemon --enable=receive-pack— the remote accepts writes.patch.txtto branches it creates itself (eval/sandbox.py:473,480).Agents run in separate containers, so a push is the only way code can cross between them.
Evidence
Measured across 117 trajectories in three
flash_10runs:git fetch teamgit push teamEvery one of those fetches returned the untouched baseline, because 0.0.22 only publishes an agent's patch at exit — by which point the peer can no longer act on it. Agents were following their instructions exactly; the instructions disabled the channel.
This prompt was the only surface that said so.
GitConnector's docstring already advertisedgit push team <branch>andgit merge team/<agent>, andagents/_coop/prompt.py(used by theclaude_code/codexadapters) already tells agents to "push so peers can fetch you". Only this config disagreed with the infrastructure it runs on.The fix
The section now describes a read/write remote and tells agents to commit and push as they work, so a colleague can read their actual diff.
The blanket prohibition is replaced with the one constraint that genuinely affects grading: the submitted
patch.txtmust contain only that agent's own changes, since the two submitted patches are merged and duplicated edits break the merge. Localfetch/merge/cherry-pickare explicitly allowed — the rule is about what you submit, not what you do in your worktree — withgit diff -- <paths>given as the way to scope a patch when both agents had to touch the same file.The rewritten section is shorter than the one it replaces (1210 vs 1713 chars), and it survives context compaction, which preserves the task message verbatim (
prefix = self.messages[:2]).Not addressed here
swe_agent/config/coop.yamlhas the same omission — it lists fetch/log/diff/dry-run and never mentions push. Left alone as a forked third-party agent; happy to include it if wanted.Verification
uv run pytest tests/— 403 passed, 63 skippedruff check/ruff format --checkcleanWhether agents actually use the channel is empirical and worth confirming with one
flash_10run countinggit push team— but 0 pushes was guaranteed while the prompt said read-only.Version bumped to 0.0.23 with a CHANGELOG entry.