Skip to content

agent: coop prompt told agents the shared git remote was read-only - #80

Merged
akhatua2 merged 2 commits into
mainfrom
fix/coop-git-prompt
Aug 6, 2026
Merged

agent: coop prompt told agents the shared git remote was read-only#80
akhatua2 merged 2 commits into
mainfrom
fix/coop-git-prompt

Conversation

@akhatua2

@akhatua2 akhatua2 commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

The bug

mini_swe_agent_v2's coop prompt titled its git section "Shared Git Remote (read-only)", listed only fetch/log/diff under "Allowed (read-only)", never mentioned push, and instructed "Do not merge, pull, or rebase their branch into yours".

Both claims are wrong:

  • GitConnector runs git daemon --enable=receive-pack — the remote accepts writes.
  • Grading never reads those branches. The evaluator applies each agent's submitted patch.txt to branches it creates itself (eval/sandbox.py:473,480).

Agents run in separate containers, so a push is the only way code can cross between them.

Evidence

Measured across 117 trajectories in three flash_10 runs:

count
git fetch team 292
git push team 0

Every one of those fetches returned the untouched baseline, because 0.0.22 only publishes an agent's patch at exit — by which point the peer can no longer act on it. Agents were following their instructions exactly; the instructions disabled the channel.

This prompt was the only surface that said so. GitConnector's docstring already advertised git push team <branch> and git merge team/<agent>, and agents/_coop/prompt.py (used by the claude_code / codex adapters) already tells agents to "push so peers can fetch you". Only this config disagreed with the infrastructure it runs on.

The fix

The section now describes a read/write remote and tells agents to commit and push as they work, so a colleague can read their actual diff.

The blanket prohibition is replaced with the one constraint that genuinely affects grading: the submitted patch.txt must contain only that agent's own changes, since the two submitted patches are merged and duplicated edits break the merge. Local fetch/merge/cherry-pick are explicitly allowed — the rule is about what you submit, not what you do in your worktree — with git diff -- <paths> given as the way to scope a patch when both agents had to touch the same file.

The rewritten section is shorter than the one it replaces (1210 vs 1713 chars), and it survives context compaction, which preserves the task message verbatim (prefix = self.messages[:2]).

Not addressed here

swe_agent/config/coop.yaml has the same omission — it lists fetch/log/diff/dry-run and never mentions push. Left alone as a forked third-party agent; happy to include it if wanted.

Verification

  • uv run pytest tests/403 passed, 63 skipped
  • ruff check / ruff format --check clean
  • YAML parses and the Jinja renders correctly for both agent ids

Whether agents actually use the channel is empirical and worth confirming with one flash_10 run counting git push team — but 0 pushes was guaranteed while the prompt said read-only.

Version bumped to 0.0.23 with a CHANGELOG entry.

The mini_swe_agent_v2 coop prompt titled its git section "Shared Git Remote
(read-only)", listed only fetch/log/diff under "Allowed (read-only)", never
mentioned push, and instructed "Do not merge, pull, or rebase their branch
into yours". Both claims are wrong.

GitConnector runs `git daemon --enable=receive-pack`, so the remote accepts
writes, and grading never reads those branches: the evaluator applies each
agent's submitted patch.txt to branches it creates itself (eval/sandbox.py).
Agents are in separate containers, so a push is the only way code can cross
between them.

Measured across 117 trajectories in three flash_10 runs: 292 `git fetch team`
and 0 `git push team`. Every fetch returned the untouched baseline, because
0.0.22 only publishes a patch at exit -- by which point the peer can no longer
act on it. Agents were following their instructions exactly.

This prompt was the only surface that said so. GitConnector's docstring
already advertised `git push team <branch>`, and agents/_coop/prompt.py (used
by the claude_code and codex adapters) already tells agents to "push so peers
can fetch you".

The section now describes a read/write remote and tells agents to commit and
push as they work. The blanket prohibition is replaced by the one constraint
that actually affects grading: the submitted patch.txt must contain only that
agent's own changes, since the two patches are merged and duplicate edits
break the merge. Local fetch/merge is explicitly allowed -- the rule is about
what you submit, not what you do in your worktree -- with `git diff -- <paths>`
given as the way to scope a patch when both agents touched the same file.

The rewritten section is shorter than the one it replaces (1210 vs 1713 chars)
and survives context compaction, which preserves the task message verbatim.
Submission was a unified diff the agent wrote to a local file. Nothing else
could see it, so the artifact that got graded was decoupled from anything a
colleague could read, and an agent could submit work it had never shared.

The agent now commits what it wants to submit, pushes its branch and opens a
PR; the PR is what gets graded. A `gh` shim implements gh pr create/list/view/
diff/checkout over the shared remote using plain git, so agents use a spelling
they already know rather than a command invented for this benchmark. A PR
tracks its branch, as on a forge: agents are told to open one early so a
colleague has context, and later commits are included once pushed.

An agent that never opens a PR submits nothing and scores zero. That is an
agent failure, and it is logged so it stays attributable rather than looking
like failed tests. Solo uses the same path against a bare repo in its own
sandbox, so there is one submission mechanism rather than two that can drift
apart -- extraction had already moved to the PR while solo.yaml still told
agents to write patch.txt, which silently discarded every solo submission.

Both prompts got smaller: patch.txt needed ~30 lines teaching a diff
incantation that exists nowhere in real engineering; gh pr create needs none.

Also fixed, all found by reviewing this change against a live run:

- Coop agents could read the whole upstream history, including commits made
  AFTER the task commit. A task image clones then checks out a sha, which
  leaves the future reachable through refs/remotes, tags, and the local branch
  the clone left at the tip -- checkout only detaches HEAD. For a task derived
  from a real PR, `git log --all -p` could show the upstream implementation of
  the feature being asked for. Setup now removes the remote, deletes those
  refs, expires the reflog and prunes, so the objects are gone rather than
  merely unreferenced.

- A submission could be re-baselined by anyone pushing to main: patches were
  diffed against a movable ref on a daemon with no access control. The base
  commit is pinned at setup.

- The modal backend decoded sandbox output as strict UTF-8, so an agent that
  read a binary file (observed: `tail .git/index`) killed its own run with a
  decode error recorded as an agent failure. It now decodes leniently, as the
  docker backend already did.

- A failed `gh` shim install was a log warning; without the shim the agent
  cannot submit at all. Setup now raises.

- The shared remote is `origin` and the clone's unreachable upstream is
  removed, so the remote agents reach for by reflex is the one that works.

Verified on real infrastructure (flask_task/5526): both agents negotiated a
shared signature over messaging, agent1 read the peer's branch with git diff,
implemented both parameters, opened a PR; 120 lines extracted, merged clean,
both feature suites passed. The same pair failed before these changes.
@akhatua2
akhatua2 merged commit 3081445 into main Aug 6, 2026
3 checks passed
@akhatua2
akhatua2 deleted the fix/coop-git-prompt branch August 6, 2026 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant