agent: coop agents could not tell a finished peer from a silent one

CHANGELOG.md

@@ -7,6 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.0.22] - 2026-08-04
+
+### Fixed
+
+- **A coop agent that finished could not be distinguished from one that was ignoring you.** When an agent submitted and exited, its peer kept talking to a mailbox nobody would ever read again: `MessagingConnector.send` queued into Redis and returned success unconditionally, with no liveness check. Measured over 8 `flash_10` pairs, **16 messages sent / 11 delivered (31% lost)**; in `pallets_click_task/2800/f1_f7` all 3 were lost — sent 35s after the peer's final turn. The sender was told `Message sent to agent2` (returncode 0) each time, and its own summary recorded *"Coordination with agent2 is ongoing … no expected conflicts"* immediately before submitting a patch that merge-conflicted. `send()` now returns `False` for a departed peer and the agent receives a non-zero result naming the cause and the recovery path, rather than a false success.
+
+- **`send_message --wait` was documented but never implemented.** `DefaultAgent._handle_send_message` guarded on `hasattr(self.comm, "send_and_wait")`, and `MessagingConnector` had no such method — so `--wait` silently degraded to fire-and-forget. Agents asking a blocking question got an instant "Message sent" and moved on. `send_and_wait(recipient, content, timeout)` now exists, returns `(delivered, replies)`, and ends the wait as soon as the peer replies **or exits**, instead of burning the full 60s on a reply that cannot arrive.
+
+- **`team/<peer>` never contained the peer's work.** `GitConnector.setup` pushes the base commit once and nothing updates the branch afterwards, while the prompt presents that remote as the sanctioned way to see a colleague's code. Across 19 agent runs there were **zero pushes**: in one pair an agent ran 24 `git fetch` / `git diff team/agent2` commands and saw the untouched baseline every time, then asked its colleague "have you submitted your branch yet?". Each agent now publishes its **submitted patch** (`patch.txt`, the artifact that is actually evaluated — not the working tree, which may differ) to `team/<agent_id>` on exit. It is built in a detached worktree from the pristine base, so the agent's own branch, index and working tree are untouched and a patch is never double-applied when the agent had already committed its work.
+
+### Added
+
+- Peers are told once, in context, when a colleague finishes: `[agent2 has completed their work and exited]`, with the branch to reconcile against. Publication is best-effort, so the exit marker records whether the patch actually reached the remote and the agent is only pointed at that branch when it really holds their submission.
+
+### Changed
+
+- The coop prompt now states what `team/<peer>` holds and when ("stays at the repository's starting state until your colleague submits" — an empty diff early means *not submitted yet*, not *no changes*), that `--wait` can return early, and that a colleague may exit before you do.
+
+## [0.0.21] - 2026-08-04
+
+### Fixed
+
+- **`capture_token_ids` (0.0.20) never produced a capture.** litellm rebuilds each response into its own model; top-level extras survive on `ModelResponse`, but extras on a *choice* are swept into `provider_specific_fields` by `convert_to_model_response_object` — which is where vLLM's per-choice `token_ids` lands. Reading the raw key yielded nothing, and a live run logged the "server returned no token ids" warning on all 816 calls while the server was returning them the whole time. Also documents that this requires an OpenAI-style provider prefix: litellm's Anthropic path uses a different transform that discards both `prompt_token_ids` and `token_ids` outright.
+
+## [0.0.20] - 2026-08-04
+
+### Added
+
+- **Optional `capture_token_ids` on `LitellmModelConfig`** (default `False`, so existing runs are byte-identical). When set, requests carry `extra_body={"return_token_ids": true}` and the ids the server actually used are stored on each assistant message as `extra["token_capture"]`, persisting into the saved trajectory. Reconstructing them afterwards is not equivalent: BPE is non-injective, tool-call serialization can differ between inference and training, and under context compaction the prompt a turn saw no longer exists in the final message list. Requires vLLM >= 0.10.2 or SGLang with `return_token_ids` support.
+
 ## [0.0.19] - 2026-05-25
 
 ### Fixed

src/cooperbench/__about__.py

@@ -1,3 +1,3 @@
 """Version information for CooperBench."""
 
-__version__ = "0.0.21"
+__version__ = "0.0.22"

src/cooperbench/agents/mini_swe_agent_v2/agents/default.py

@@ -16,6 +16,10 @@
 from cooperbench.agents.mini_swe_agent_v2.exceptions import InterruptAgentFlow, LimitsExceeded
 from cooperbench.agents.mini_swe_agent_v2.utils.serialize import recursive_merge
 
+# Name of the shared git remote created by GitConnector.  Referenced in the messages the
+# agent sees, so it must match GitConnector.REMOTE_NAME.
+GIT_REMOTE = "team"
+
 
 class AgentConfig(BaseModel):
     """Check the config files in config/ for example settings."""
@@ -197,9 +201,74 @@ def run(self, task: str = "", **kwargs) -> dict:
             finally:
                 self.save(self.config.output_path)
             if self.messages[-1].get("role") == "exit":
+                published = self._publish_final_work()
+                if self.comm:
+                    self.comm.mark_exited(published=published)
                 break
         return self.messages[-1].get("extra", {})
 
+    def _publish_final_work(self) -> bool:
+        """Publish this agent's *submitted patch* to the shared remote before exiting.
+
+        The remote is seeded with the base commit at setup and never updated again, so a
+        peer inspecting ``{GIT_REMOTE}/<agent>`` sees the untouched baseline no matter how
+        much work was done — and reasonably concludes their colleague has not started.
+        The prompt presents that remote as the sanctioned way to see a colleague's code,
+        so today it is a documented capability that silently returns nothing.
+
+        We publish ``patch.txt``, not the working tree: patch.txt is the artifact that
+        gets evaluated and merged, and agents are free to submit a subset of their edits.
+        Showing a peer the tree would show them something other than what will be merged.
+
+        Built in a detached worktree so the agent's own branch, index and working tree
+        are untouched — the adapter still reads patch.txt from the container afterwards.
+
+        Returns True only when the patch actually reached the remote.  Peers are told to
+        read that branch, so a failed publication must not be reported as a success.
+        Best-effort otherwise: publication must never fail a run.
+        """
+        if not getattr(self, "comm", None):
+            return False  # solo run: nobody to publish to
+        agent_id = self.comm.agent_id
+        script = (
+            "set -e; "
+            # the repo is normally /workspace/repo, but fall back rather than guess
+            'repo=/workspace/repo; [ -d "$repo/.git" ] || repo=/workspace; cd "$repo"; '
+            # No patch means nothing to publish.  Exit non-zero so this is NOT reported as
+            # a successful publication -- claiming the branch holds their submission when
+            # it holds the baseline is the exact failure this change removes.
+            'test -s patch.txt || { echo "no patch.txt to publish"; exit 3; }; '
+            # Branch from the pristine base, not from HEAD: the evaluator applies patch.txt
+            # to the base, so mirroring that is what makes the branch equal the submission.
+            # If the agent committed its work, HEAD already contains it and applying the
+            # patch on top would double-apply.  setup() seeds the remote's main with base.
+            f"git fetch -q {GIT_REMOTE} 2>/dev/null || true; "
+            f"base=$(git rev-parse {GIT_REMOTE}/main 2>/dev/null "
+            "|| git rev-parse main 2>/dev/null || git rev-parse HEAD); "
+            'tmp=$(mktemp -d); git worktree add -q --detach "$tmp" "$base"; '
+            'cp patch.txt "$tmp/.__submitted.patch"; cd "$tmp"; '
+            "git apply --ignore-whitespace .__submitted.patch || git apply --3way .__submitted.patch; "
+            "rm -f .__submitted.patch; git add -A; "
+            f"git -c user.email=agent@cooperbench -c user.name={agent_id} "
+            f'commit -q --allow-empty -m "submitted work by {agent_id}"; '
+            f"git push -q -f {GIT_REMOTE} HEAD:refs/heads/{agent_id}; "
+            # `worktree remove` matches on git's own resolved path, which differs from
+            # $tmp wherever the temp dir sits behind a symlink; deleting the directory and
+            # pruning is equivalent and does not depend on that matching.
+            'cd /; rm -rf "$tmp"; git -C "$repo" worktree prune 2>/dev/null || true'
+        )
+        try:
+            result = self.env.execute({"command": script})
+            if (result or {}).get("returncode") == 0:
+                self.log(f"PUBLISHED submitted patch to {GIT_REMOTE}/{agent_id}")
+                return True
+            self.logger.warning(
+                f"could not publish to {GIT_REMOTE}/{agent_id}: {(result or {}).get('output', '')[:300]}"
+            )
+        except Exception as e:  # noqa: BLE001 - never fail a run over publication
+            self.logger.warning(f"could not publish to {GIT_REMOTE}/{agent_id}: {e}")
+        return False
+
     def step(self) -> list[dict]:
         """Query the LM, execute actions. Polls for inter-agent messages
         and (in team mode) the shared task list before querying."""
@@ -215,6 +284,7 @@ def step(self) -> list[dict]:
                         content=f"[Message from {msg['from']}]: {msg['content']}",
                     )
                 )
+            self._announce_departed_peers()
         # In team mode, also refresh the shared task list so the LLM
         # sees the live state of who's working on what before its next
         # response.  ``team_poller`` is set by the adapter when team
@@ -351,31 +421,111 @@ def execute_actions(self, message: dict) -> list[dict]:
             outputs.append(self.env.execute(action))
         return self.add_messages(*self.model.format_observation_messages(message, outputs, self.get_template_vars()))
 
+    def _announce_departed_peers(self) -> None:
+        """Tell the agent, once, when a peer has finished and left.
+
+        Otherwise an agent can spend the rest of its run waiting for an answer from a
+        colleague who is no longer running, with nothing in its context to indicate that.
+        Announced once per peer; the recovery path is named because the peer's work is
+        still reachable through git even though the conversation is over.
+        """
+        if not self.comm:
+            return
+        announced = getattr(self, "_departed_announced", None)
+        if announced is None:
+            announced = self._departed_announced = set()
+        for peer in getattr(self.comm, "agents", []) or []:
+            if peer == self.comm.agent_id or peer in announced:
+                continue
+            if not self.comm.has_exited(peer):
+                continue
+            announced.add(peer)
+            self.log(f"PEER EXITED: {peer}")
+            self.add_messages(
+                self.model.format_message(
+                    role="user",
+                    content=(
+                        f"[{peer} has completed their work and exited] They will not read or "
+                        f"answer further messages.\n\n{self._peer_work_pointer(peer)}"
+                    ),
+                )
+            )
+
+    def _peer_work_pointer(self, peer: str) -> str:
+        """Where to find a departed peer's work — only claimed when it is really there.
+
+        Publication to the shared remote is best-effort, so asserting the branch holds
+        their submission when it does not would repeat exactly the failure this whole
+        change exists to remove: telling the agent something untrue and letting it act
+        on it.
+        """
+        if self.comm and self.comm.has_published(peer):
+            return (
+                f"Their submitted patch is on branch {GIT_REMOTE}/{peer}. If your changes "
+                f"overlap theirs, inspect and reconcile before you submit:\n"
+                f"    git fetch {GIT_REMOTE} && git diff HEAD...{GIT_REMOTE}/{peer}"
+            )
+        return (
+            f"Their work could NOT be published to {GIT_REMOTE}/{peer}, so that branch does "
+            f"not reflect what they submitted — do not rely on it. Proceed on your own "
+            f"judgement and keep your changes as self-contained as you can."
+        )
+
     def _handle_send_message(self, action: dict) -> dict:
         """Handle a send_message call via the messaging connector.
 
-        ``wait=True`` (when the agent wrote ``send_message --wait ...`` in
-        bash) uses ``send_and_wait`` so the peer's reply comes back in the
-        same tool output.
+        ``wait=True`` (when the agent wrote ``send_message --wait ...`` in bash) blocks
+        until the peer replies, the peer exits, or the timeout elapses.
+
+        A send to a peer that has already finished is reported as a failure, not a
+        success.  Reporting success there is actively misleading: the agent believes it
+        has coordinated, keeps waiting for an answer that cannot arrive, and ships work
+        that was never reconciled.
         """
         recipient = action.get("recipient", "")
         content = action.get("content", "")
         wait = action.get("wait", False)
 
         if wait and hasattr(self.comm, "send_and_wait"):
-            replies = self.comm.send_and_wait(recipient, content, timeout=60)
+            delivered, replies = self.comm.send_and_wait(recipient, content, timeout=60)
+            if not delivered:
+                return self._peer_gone_result(recipient)
             self.log(f"SENT (blocking) to {recipient}: {content[:80]}...")
             self.sent_messages.append({"to": recipient, "content": content})
             output = f"Message sent to {recipient}"
-            for r in replies or []:
-                output += f"\n\n[Reply from {r['from']}]: {r['content']}"
+            if replies:
+                for r in replies:
+                    output += f"\n\n[Reply from {r['from']}]: {r['content']}"
+            elif self.comm.has_exited(recipient):
+                output += (
+                    f"\n\nNo reply: {recipient} has since completed their work and exited.\n"
+                    f"{self._peer_work_pointer(recipient)}"
+                )
             return {"output": output, "returncode": 0, "exception_info": ""}
 
-        self.comm.send(recipient, content)
+        if not self.comm.send(recipient, content):
+            return self._peer_gone_result(recipient)
         self.log(f"SENT to {recipient}: {content[:80]}...")
         self.sent_messages.append({"to": recipient, "content": content})
         return {"output": f"Message sent to {recipient}", "returncode": 0, "exception_info": ""}
 
+    def _peer_gone_result(self, recipient: str) -> dict:
+        """Tell the agent its peer is gone, and what to do about it.
+
+        Phrased as a terminal state rather than a delivery error so the agent does not
+        retry, and names the recovery path so the peer's work is still reachable.
+        """
+        self.log(f"NOT DELIVERED to {recipient}: peer already exited")
+        return {
+            "output": (
+                f"{recipient} has already completed their work and exited. Your message was "
+                f"NOT delivered and no reply will come — do not send further messages to "
+                f"them.\n\n{self._peer_work_pointer(recipient)}"
+            ),
+            "returncode": 1,
+            "exception_info": "",
+        }
+
     def serialize(self, *extra_dicts) -> dict:
         """Serialize agent state to a json-compatible nested dictionary for saving."""
         last_message = self.messages[-1] if self.messages else {}

src/cooperbench/agents/mini_swe_agent_v2/config/coop.yaml

@@ -50,18 +50,23 @@ agent:
     MSG
     ```
 
+    `--wait` returns as soon as they reply, or immediately if they have already finished — it does not always burn the full 60 seconds.
+
     Tips:
     - A `send_message` by itself is a valid tool call. You do not need to combine it with other commands.
     - You can freely choose `--wait` or regular `send_message` based on what the situation requires.
     - Messages from your colleague appear as: [Message from ...]: ...
     - IMPORTANT: Only the system delivers messages from your colleague. Never write `[Message from ...]` yourself.
+    - Your colleague may finish and exit before you do. When that happens you are told so explicitly, and any further `send_message` to them fails with a non-zero exit code instead of being silently discarded. Do not keep messaging or waiting after that — reconcile against their published branch and submit.
     {% endif %}
 
     {% if git_enabled %}
     ## Shared Git Remote (read-only)
 
     A shared remote called `team` lets you see your colleague's actual code changes. Use communication as your primary way to coordinate. Use git only to peek at your colleague's code in complex cases or to verify merging towards the end. Think about how experienced software engineers would make the most of direct communication and git at various stages — and do that.
 
+    **What that branch holds, and when.** `team/{{ agents | reject('equalto', agent_id) | first }}` stays at the repository's starting state until your colleague submits; their patch is published there automatically at that moment. So an empty diff early on means "they have not submitted yet" — it does NOT mean they have made no changes. Ask them directly rather than inferring from git. You are told explicitly when they finish.
+
     Your patches are merged automatically after you both submit. Do not merge, pull, or rebase their branch into yours, as the automatic merge will see duplicate changes and fail.
 
     Allowed (read-only):