diff --git a/.gitignore b/.gitignore index ab18eaa2..0b6ee368 100644 --- a/.gitignore +++ b/.gitignore @@ -63,3 +63,6 @@ e2e/coverage-report.json node_modules/ docs/.vitepress/cache/ docs/.vitepress/dist/ + +# Local verification project +/verify-multi-turn/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f147964..3f15773b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,21 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [0.4.0] - 2026-07-07 + +### Added +- **Multi-turn conversation evaluation** (`input.turns`): define sequential + user turns with per-turn `post_condition` gates (`must_contain_all`, + `must_contain_any`, `must_not_contain`, `on_fail: fail|skip_remaining`) and + `capture` for regex-based variable extraction between turns. +- Per-turn judge assertions: `turn_response_contains`, + `turn_response_not_contains`, `tool_called_in_turn`, + `tool_not_called_in_turn` — all scoped by turn number. +- Session resumption support for `claude_code` (`--resume`), `qodercli` + (`-r `), and `codex` (`codex resume `) engines; + unsupported engines fall back to batch mode. +- `TurnResults` field in JSON, HTML, and JUnit reports (`turn_results` in + `result.json`; turn-scoped failure details in JUnit XML). ## [0.3.0] - 2026-07-03 diff --git a/docs/design/custom-engine.md b/docs/design/custom-engine.md index c759ea62..e67fe115 100644 --- a/docs/design/custom-engine.md +++ b/docs/design/custom-engine.md @@ -464,6 +464,14 @@ The `local` transport runs a command via `runtime.Exec`. The command runs inside current runtime, so it can access the runtime workspace, installed skills, fixtures, MCP config, and environment variables. +Top-level `skills` are installed before the main run agent executes. If +`judge.type: agent_judge` declares `judge.skills`, those Skills are installed +separately before the judge agent runs. Both paths use the Agent adapter's +`InstallSkill` implementation; skill-up does not concatenate Skill files into +the prompt as a fallback. Custom and remote engines that need judge Skills must +make their agent process discover the installed Skill directory according to +their own conventions. + Example: ```yaml diff --git a/docs/design/multi-turn-conversation-implementation-plan.md b/docs/design/multi-turn-conversation-implementation-plan.md new file mode 100644 index 00000000..c60f2814 --- /dev/null +++ b/docs/design/multi-turn-conversation-implementation-plan.md @@ -0,0 +1,555 @@ +# Multi-Turn Conversation Evaluation Implementation Plan + +Status: draft for review +Source proposal: SUP-0001 Multi-Turn Conversation Evaluation Support +Last reviewed: 2026-07-03 + +## Objective + +Implement deterministic, scripted multi-turn conversation evaluation in +skill-up. A case using `input.turns` must execute turn by turn, preserve one +agent session across turns, evaluate intermediate `post_condition` checks, +capture values for later prompts, expose per-turn results to judges and +reports, and keep existing `input.prompt` cases behavior-compatible. + +The implementation should keep module ownership clear: + +- `internal/config`: schema and validation only. +- `internal/agent`: agent execution and session-resume adapters only. +- `internal/evaluator`: turn orchestration, runtime lifecycle, aggregation. +- `internal/judge`: assertions over already-collected facts. +- `internal/report` and `internal/runner`: serialization and presentation of + evaluation facts. + +## Current State + +The repository already contains important foundations: + +- `internal/config.Input` already has `Prompt` and `Turns`. +- `internal/config.Turn` already has `Role`, `Content`, and `PostCondition`. +- `pkg/transcript.Message` already carries `Turn`. +- `internal/agent.Agent.Run` already accepts `[]transcript.Message`. +- `agent.SessionResult` already carries transcript, tokens, final message, and + artifacts. +- Custom engines already receive a structured `SessionInput.messages` contract. + +The missing core is that evaluator execution still sends all turns to one +`agent.Run` call. Built-in CLI agents then collapse those messages through +`BuildInstructionFromMessages`, so there is no real turn-by-turn interaction, +no intermediate condition checking, and no per-turn judge input. + +## Architecture + +```mermaid +flowchart LR + C["config loader + validator"] --> E["evaluator"] + E -->|single-turn| A["agent.Run"] + E -->|multi-turn first turn| A + E -->|multi-turn next turns| R["agent.SessionResumer.RunTurn"] + A --> TR["TurnResult aggregation"] + R --> TR + TR --> J["judge.Input + rule_based assertions"] + J --> RP["report.Input / result.json / HTML / JUnit"] +``` + +Multi-turn execution will be a new branch inside the existing case lifecycle, +after runtime preparation and before the current single-call `agent.Run`. The +single-turn path remains the existing path. + +Key data flow: + +1. Validator accepts only well-formed `input.turns`, post conditions, capture + rules, and per-turn judge rules. +2. Evaluator starts a case runtime exactly once. +3. Turn 1 calls `Agent.Run` to create a real agent session. +4. Later turns call `SessionResumer.RunTurn` with the extracted session ID. +5. Each turn produces an evaluator-owned `TurnResult`. +6. Evaluator aggregates final `SessionResult`, transcript, status, and judge + input. +7. Judge evaluates global and per-turn assertions without knowing agent details. +8. Runner/report serialize turn results as part of the existing result model. + +## Module Plan + +### 1. Config Schema + +Files: + +- `internal/config/schema.go` +- `internal/config/validator.go` +- `internal/config/schema_test.go` +- `internal/config/validator_test.go` +- `internal/config/defaults.yaml` if examples/default docs need updates + +Add schema fields: + +- `Turn.Capture []CaptureRule` +- `Turn.TimeoutSeconds int` +- `PostCondition.MustContainAll []string` +- `PostCondition.MustNotContain []string` +- `CaptureRule.Variable string` +- `CaptureRule.Pattern string` +- `CaptureRule.JSONPath string` +- `Rule.TurnResponseContains *TurnResponseContainsRule` +- `Rule.TurnResponseNotContains *TurnResponseNotContainsRule` +- `Rule.ToolCalledInTurn *ToolCalledInTurnRule` +- `Rule.ToolNotCalledInTurn *ToolNotCalledInTurnRule` + +Validation rules: + +- A case must use exactly one meaningful input mode: + `input.prompt` or `input.turns`. If both are present, return a validation + error to avoid ambiguous execution. +- `input.turns[*].role` must be `user` for this phase. Other roles are not + needed for scripted user turns and make resume semantics ambiguous. +- `input.turns[*].content` must not be empty. +- `post_condition.on_fail` must be empty, `fail`, or `skip_remaining`; empty + means `fail`. +- A post condition must include at least one of `must_contain_any`, + `must_contain_all`, or `must_not_contain`. +- `capture[*].variable` must match a conservative identifier pattern such as + `^[A-Za-z_][A-Za-z0-9_]*$`. +- Each capture rule must specify exactly one extractor: `pattern` or + `jsonpath`. +- Regex patterns must compile during validation. +- `timeout_seconds` must be non-negative. +- Per-turn judge rules must reference turn numbers >= 1 and, when total turns + are known, <= `len(input.turns)`. +- `turn_response_contains` must specify `contains_all` or `contains_any`. +- `turn_response_not_contains.not_contains` is required. +- `tool_called_in_turn.name` and `tool_not_called_in_turn.name` are required. + +Design constraint: validation must not know about agent implementations or +runtime behavior. + +### 2. Agent Resume Boundary + +Files: + +- `internal/agent/agent.go` +- `internal/agent/claude_code.go` +- `internal/agent/qodercli.go` +- `internal/agent/codex.go` +- `internal/agent/*_test.go` + +Add a small optional interface without changing `Agent`: + +```go +type SessionResumer interface { + RunTurn(ctx context.Context, rt Runtime, opts ExecOptions, message transcript.Message, sessionID string) (*SessionResult, error) +} +``` + +Add `SessionResult.SessionID string` so evaluator can resume without +type-specific parsing. + +Responsibilities: + +- Agent adapters know how to start/resume their own CLI sessions. +- Agent adapters normalize session IDs into `SessionResult.SessionID`. +- Evaluator only checks `SessionResumer` and passes the ID forward. + +Implementation order: + +1. `claude_code`: first turn already generates a UUID and passes + `--session-id`. Populate `SessionResult.SessionID` from that generated ID + even when output parsing does not return it. Add `RunTurn` using + `claude --resume -p`. +2. `qodercli`: switch or add a JSON-output path that can parse `sessionId`. + Add `RunTurn` using exact `-r ` rather than "continue latest". +3. `codex`: implement only after safe session correlation is solved. Prefer a + CLI-emitted session ID. If unavailable, use a case-isolated session + directory or a locked before/after diff of the session directory. Do not use + global "latest session" lookup under concurrent `cases.parallelism`. +4. `custom`: no immediate `SessionResumer`. Custom engines already accept full + message history in one request, so they remain fallback-capable until a + future custom resume contract is designed. + +Boundary cases: + +- Empty `sessionID` after turn 1 with more turns remaining becomes a clear case + execution error. +- Resume command auth/rate-limit handling should reuse existing first-run + signal detection. +- `ExecOptions.ArtifactDir`, timeout, env, model, workspace, and observability + metadata must be honored consistently for first and resumed turns. + +### 3. Evaluator Multi-Turn Engine + +Files: + +- `internal/evaluator/evaluator.go` +- likely new focused helper file, for example `internal/evaluator/multiturn.go` +- `internal/evaluator/evaluator_test.go` + +Add evaluator-owned types: + +```go +type TurnStatus string + +const ( + TurnCompleted TurnStatus = "completed" + TurnSkipped TurnStatus = "skipped" + TurnFailed TurnStatus = "failed" + TurnError TurnStatus = "error" +) + +type TurnResult struct { + TurnNumber int + Content string + Response string + Transcript transcript.Transcript + SessionResult *agent.SessionResult + Status TurnStatus + Reason string + CapturedVars map[string]string +} +``` + +Execution behavior: + +- Branch to multi-turn only when `len(input.turns) > 1`. +- Keep existing single-turn path untouched for `input.prompt` and one-turn + `input.turns`. +- Prepare runtime, skills, MCP, workspace diff hooks, artifact dirs, and judge + config once per case, then execute all turns in that same runtime. +- Turn 1 calls `runAgent.Run` with a single user message. +- Turns 2..N call `resumer.RunTurn`. +- Each turn may have a narrower `turn.timeout_seconds`; the overall case + timeout still bounds the whole case. +- After each turn, evaluate `post_condition`. +- If `post_condition` fails with `on_fail: fail`, mark the case `FAIL` and do + not execute later turns. +- If it fails with `on_fail: skip_remaining`, mark remaining turns skipped and + return a `SKIP` only when the scenario produced no judgeable completed path. + If there are completed turns plus skipped later turns, judge should still run + when configured and per-turn rules decide the final status. +- Capture variables after a successful post condition. +- Template substitution happens immediately before sending a turn. Unresolved + `{{variable}}` placeholders are `ERROR`, not raw prompt text sent to the + agent. + +Result aggregation: + +- Preserve every user turn and assistant response in the final transcript with + correct turn numbers. +- Prefer per-turn transcripts from adapters; when an adapter returns a + cumulative transcript, normalize/deduplicate before appending. +- Final message is the last completed turn response. +- `Turns` is the number of completed agent turns. +- Token counts sum across turns when adapters return per-turn usage. +- Artifact handling must still run for errors/timeouts, matching current + single-turn behavior. +- `Expect` and existing global judge rules evaluate against the final aggregate + session. +- Per-turn judge rules receive `judge.Input.TurnResults`. + +Avoid recursion trap: fallback must not call `executeCaseOnce` in a way that +re-enters the multi-turn branch forever. Implement fallback through a dedicated +single-shot helper or a boolean execution mode guard. + +### 4. Post Conditions and Capture + +Files: + +- `internal/evaluator/multiturn.go` +- `internal/evaluator/multiturn_test.go` + +Post-condition semantics: + +- `must_contain_all`: all required strings must appear. +- `must_contain_any`: at least one string must appear. +- `must_not_contain`: none may appear. +- Matching should initially follow existing `output_contains` behavior: + case-sensitive by default. If case-insensitive behavior is desired, make it + explicit in a later schema extension rather than silently diverging. +- Empty `on_fail` means `fail`. +- Failure reason must name the missing or forbidden keyword for debugability. + +Capture semantics: + +- Regex capture supports named group `(?P...)`; if absent, allow exactly + one capture group as a convenience. +- No match, invalid group, or empty value is an execution `ERROR`. +- JSONPath capture is phase 3. Keep phase 1/2 regex-only if avoiding a new + dependency is important. +- Captured values are stored in an evaluator-local map scoped to one case + execution only. +- Variables are never shared across cases, baseline variants, retries, or + iterations. +- If a later turn references an unknown variable, fail before invoking the + agent. + +### 5. Judge Per-Turn Assertions + +Files: + +- `internal/judge/judge.go` +- `internal/judge/rule_based.go` +- `internal/judge/rule_based_test.go` + +Add judge-visible turn result: + +```go +type TurnResult struct { + TurnNumber int `json:"turn_number"` + Content string `json:"content"` + Response string `json:"response"` + Transcript transcript.Transcript `json:"transcript"` + Status string `json:"status"` + Reason string `json:"reason,omitempty"` +} +``` + +Add `TurnResults []TurnResult` to `judge.Input`. + +Rules: + +- `turn_response_contains`: check one completed turn response with + `contains_all` and/or `contains_any`. +- `turn_response_not_contains`: check forbidden strings are absent from one + completed turn response. +- `tool_called_in_turn`: check tool calls in that turn transcript, with the same + partial argument matching semantics as `tool_called`. +- `tool_not_called_in_turn`: check absence in that turn transcript. + +Failure behavior: + +- Missing turn is a failing assertion for positive checks. +- Missing turn should also fail for `tool_not_called_in_turn` in this + implementation, because validation should already reject impossible turn + numbers and a missing executed turn usually means the scenario did not run as + expected. +- A skipped, failed, or errored turn is not "completed"; response assertions + against it fail with evidence naming the status. + +### 6. Reporting and Artifacts + +Files: + +- `internal/report/reporter.go` +- `internal/report/html.go` +- `internal/report/junit.go` if JUnit should include turn details in failure text +- `internal/runner/runner.go` +- report tests + +Add `TurnResults` to `evaluator.EvalResult` and `report.CaseResult`. + +Reporting behavior: + +- `result.json` and `report.json` include `turn_results`. +- `response.md` remains the final response for compatibility. +- Add a turn transcript artifact when `report.artifacts` includes `transcript`, + using the existing artifact flow where possible. +- HTML should show per-turn prompt, response, status, and reason in a compact + case detail section. +- JUnit can remain case-level, but assertion evidence should include turn + numbers so CI failures are actionable. + +Compatibility rule: existing report fields keep their meaning. New fields are +optional and omitted for single-turn cases. + +### 7. Documentation and Examples + +Files: + +- `docs/guide/writing-evals.md` +- `docs/zh/guide/writing-evals.md` +- `skills/skill-upper/references/case-yaml.md` +- `skills/skill-upper/assets/case.yaml.tmpl` +- `e2e/testdata/multi-turn-conversation/*` +- `CHANGELOG.md` when implementation is user-visible + +Docs should explain: + +- When to use `input.prompt` versus `input.turns`. +- `post_condition` as an inter-turn gate, not a replacement for judge. +- Capture/template syntax and failure behavior. +- Agent support matrix and fallback behavior. +- Per-turn assertion examples for phase gating, confirmation, clarification, + iterative refinement, and cross-turn references. + +Update the existing `e2e/testdata/multi-turn-conversation` cases so they use +real `input.turns`, not a single prompt describing fake turns. + +## Phased Delivery + +### Phase 0: Safety Harness + +Goal: prepare tests and seams before behavior changes. + +- Add unit tests documenting current single-turn compatibility. +- Add fake `SessionResumer` agent in evaluator tests. +- Add report fixture expectations for optional `turn_results`. + +Validation: + +- `go test -race ./internal/config ./internal/evaluator ./internal/judge ./internal/report` + +### Phase 1: Real Sequential Multi-Turn Core + +Goal: run scripted `input.turns` sequentially for agents that support resume. + +- Add `SessionResult.SessionID`. +- Add `SessionResumer`. +- Implement `claude_code` resume first. +- Add evaluator multi-turn branch, turn loop, post conditions, aggregation. +- Add fallback mode with explicit warning and no recursive branch. +- Add JSON result `turn_results`. + +Validation: + +- Unit tests for happy path, `post_condition fail`, + `post_condition skip_remaining`, empty session ID, turn timeout, and fallback. +- Existing single-turn evaluator tests pass unchanged. + +### Phase 2: Per-Turn Judge Assertions + +Goal: let rule-based judge assert specific turn responses and tool calls. + +- Add per-turn rule structs and validation. +- Add `judge.Input.TurnResults`. +- Implement four per-turn rule evaluators. +- Add tests for out-of-range, skipped turn, failed turn, positive and negative + matching, and tool args. + +Validation: + +- `go test -race ./internal/config ./internal/judge ./internal/evaluator` + +### Phase 3: Capture and Template Variables + +Goal: support deterministic cross-turn value passing. + +- Add regex capture. +- Add template rendering with unresolved-variable detection. +- Decide whether JSONPath ships now. If yes, add dependency through `go get` + and document license; if no, keep `jsonpath` validation rejected until a + follow-up. +- Add tests for regex named group, one unnamed group, no match, empty capture, + invalid variable, and unresolved placeholder. + +Validation: + +- `go test -race ./internal/config ./internal/evaluator` + +### Phase 4: Broader Agent Support and E2E + +Goal: make built-in agent support practical and covered. + +- Implement `qodercli` exact-session resume. +- Implement `codex` only with safe session correlation. If not safe, return an + explicit unsupported multi-turn error for codex rather than risk cross-case + contamination. +- Convert `e2e/testdata/multi-turn-conversation` to true multi-turn cases. +- Add or update E2E tests that can run with mock/custom agent support where + possible, and keep real-agent E2E guarded like existing engine tests. + +Validation: + +- `make test` +- `make verify` +- `make e2e` if `e2e/` or `internal/runner/` behavior is touched. + +## Testing Matrix + +Unit tests: + +- Config parses all new fields. +- Config rejects ambiguous prompt+turns. +- Config rejects invalid roles, empty content, invalid post conditions, + invalid capture rules, invalid regex, invalid timeout, and bad per-turn judge + rules. +- Evaluator executes turns in order and passes the same session ID to resumed + turns. +- Evaluator stops on `post_condition fail`. +- Evaluator skips remaining turns on `skip_remaining`. +- Evaluator handles missing session ID after first turn. +- Evaluator respects per-turn timeout inside the case timeout. +- Evaluator aggregates transcript, tokens, turns, final message, and errors. +- Capture/template rendering succeeds and fails with clear diagnostics. +- Judge per-turn response assertions pass/fail correctly. +- Judge per-turn tool assertions pass/fail correctly. +- Reports omit `turn_results` for single-turn cases and include them for + multi-turn cases. + +Integration tests: + +- Multi-turn case with fake resumer passes through full evaluator + judge. +- Multi-turn fallback produces a warning/result mode and per-turn assertions + fail clearly. +- Baseline mode keeps separate sessions for `with_skill` and `without_skill`. +- Retry re-runs the whole case with a fresh runtime/session and empty capture + variables. +- Workspace artifacts are collected even when a later turn errors. + +E2E tests: + +- Phase gate skip attempt: turn 1 enters phase, turn 2 tries to skip, judge + asserts turn 2 rejection. +- Double confirmation: turn 1 asks confirmation, turn 2 executes only after + confirmation. +- Capture/reference: turn 1 emits an identifier, turn 2 references it via + `{{variable}}`. +- Report verification: `result.json`, `report.json`, and HTML include turn + details and complete transcript. + +Manual validation: + +- Run a real `claude_code` multi-turn case with `cases.parallelism: 1`. +- Run with `cases.parallelism: 2` to verify sessions do not cross. +- Run an unsupported agent/custom engine to verify fallback or explicit + unsupported diagnostics. + +Required final gates before declaring implementation complete: + +```bash +make fmt +make verify +make test +make e2e # required if e2e/ or internal/runner/ changes are included +``` + +## Edge Cases and Decisions + +- `input.prompt` plus `input.turns`: reject during validation. +- Single `input.turns` item: use existing single-turn path unless a + post-condition/capture is present. If post-condition/capture is present, use + multi-turn machinery so those fields are honored. +- Unsupported resume: default to explicit fallback only when the case has no + `post_condition`, no `capture`, and no per-turn judge rules. Otherwise return + `ERROR` because fallback cannot satisfy the declared semantics. +- Cumulative transcripts from resumed agents: normalize by turn number and + avoid duplicate prior messages in aggregate transcript. +- Tool calls without turn numbers: assign the current turn when they are part + of a per-turn `RunTurn` result. +- Agent returns empty response: post-condition and judge rules fail normally; + empty final response without error remains valid for cases with no response + assertions. +- Turn timeout: mark case `ERROR`, preserve partial turn result, collect + artifacts. +- Case timeout: current timeout annotation remains the outer authority. +- Capture value contains braces: treat as plain text; do not recursively render + captured values. +- Capture variable collision: later captures may overwrite earlier variables, + but the turn result records what each turn captured. Consider validator + warning only if duplicate variables become confusing in practice. +- Baseline mode: `with_skill` and `without_skill` must never share session IDs + or capture variables. +- Parallel cases: never use "latest session" APIs for resume. All session + correlation must be exact. +- Reports and grading: `post_condition fail` is a case `FAIL`; infrastructure + errors such as invalid template, capture failure, resume failure, and timeout + are `ERROR`; scenario not applicable due to `skip_remaining` may be `SKIP`. + +## Review Questions + +- Should unsupported agents use single-shot fallback broadly, or should + fallback be allowed only when the case does not declare post conditions, + capture, or per-turn assertions? +- Should per-turn string matching be case-sensitive like existing + `output_contains`, or should new per-turn rules be case-insensitive? +- Should JSONPath capture be included in the first implementation, or shipped + after regex capture is stable? +- Should one-turn `input.turns` with `post_condition` use multi-turn machinery + immediately, even though it is only one turn? diff --git a/docs/design/multi-turn-conversation-implementation-plan.zh.md b/docs/design/multi-turn-conversation-implementation-plan.zh.md new file mode 100644 index 00000000..6554397c --- /dev/null +++ b/docs/design/multi-turn-conversation-implementation-plan.zh.md @@ -0,0 +1,445 @@ +# 多轮对话评估实现计划 + +状态:待评审草案 +来源提案:SUP-0001 多轮对话评估支持 +最后评审:2026-07-03 + +## 目标 + +在 skill-up 中实现确定性的、脚本化的多轮对话评估。使用 `input.turns` 的测试用例必须逐轮执行,在轮次间保持同一个 agent 会话,评估中间的 `post_condition` 检查,捕获值供后续提示使用,将每轮结果暴露给 judge 和报告,并保持现有 `input.prompt` 用例的向后兼容。 + +实现应保持清晰的模块职责: + +- `internal/config`:仅负责 schema 和验证。 +- `internal/agent`:仅负责 agent 执行和会话恢复适配器。 +- `internal/evaluator`:轮次编排、运行时生命周期、结果聚合。 +- `internal/judge`:对已收集事实的断言。 +- `internal/report` 和 `internal/runner`:评估事实的序列化和展示。 + +## 当前状态 + +仓库已包含重要基础: + +- `internal/config.Input` 已有 `Prompt` 和 `Turns`。 +- `internal/config.Turn` 已有 `Role`、`Content` 和 `PostCondition`。 +- `pkg/transcript.Message` 已携带 `Turn`。 +- `internal/agent.Agent.Run` 已接受 `[]transcript.Message`。 +- `agent.SessionResult` 已携带 transcript、tokens、final message 和 artifacts。 +- 自定义引擎已接收结构化的 `SessionInput.messages` 契约。 + +缺失的核心是 evaluator 执行仍将所有轮次发送给一次 `agent.Run` 调用。内置 CLI agent 随后通过 `BuildInstructionFromMessages` 折叠这些消息,因此没有真正的逐轮交互、没有中间条件检查、也没有每轮 judge 输入。 + +## 架构 + +```mermaid +flowchart LR + C["config loader + validator"] --> E["evaluator"] + E -->|single-turn| A["agent.Run"] + E -->|multi-turn first turn| A + E -->|multi-turn next turns| R["agent.SessionResumer.RunTurn"] + A --> TR["TurnResult aggregation"] + R --> TR + TR --> J["judge.Input + rule_based assertions"] + J --> RP["report.Input / result.json / HTML / JUnit"] +``` + +多轮执行将是现有用例生命周期中的新分支,在运行时准备之后、当前单次调用 `agent.Run` 之前。单轮路径保持现有路径不变。 + +关键数据流: + +1. 验证器仅接受格式良好的 `input.turns`、post conditions、capture 规则和每轮 judge 规则。 +2. Evaluator 为每个用例精确启动一次用例运行时。 +3. 第 1 轮调用 `Agent.Run` 创建真实的 agent 会话。 +4. 后续轮次调用 `SessionResumer.RunTurn`,传入提取的会话 ID。 +5. 每轮产生 evaluator 拥有的 `TurnResult`。 +6. Evaluator 聚合最终 `SessionResult`、transcript、状态和 judge 输入。 +7. Judge 评估全局和每轮断言,无需了解 agent 细节。 +8. Runner/report 将轮次结果作为现有结果模型的一部分进行序列化。 + +## 模块计划 + +### 1. Config Schema + +文件: + +- `internal/config/schema.go` +- `internal/config/validator.go` +- `internal/config/schema_test.go` +- `internal/config/validator_test.go` +- `internal/config/defaults.yaml`(如示例/默认文档需要更新) + +添加 schema 字段: + +- `Turn.Capture []CaptureRule` +- `Turn.TimeoutSeconds int` +- `PostCondition.MustContainAll []string` +- `PostCondition.MustNotContain []string` +- `CaptureRule.Variable string` +- `CaptureRule.Pattern string` +- `CaptureRule.JSONPath string` +- `Rule.TurnResponseContains *TurnResponseContainsRule` +- `Rule.TurnResponseNotContains *TurnResponseNotContainsRule` +- `Rule.ToolCalledInTurn *ToolCalledInTurnRule` +- `Rule.ToolNotCalledInTurn *ToolNotCalledInTurnRule` + +验证规则: + +- 用例必须使用恰好一种有意义的输入模式:`input.prompt` 或 `input.turns`。如果两者都存在,返回验证错误以避免歧义执行。 +- `input.turns[*].role` 必须为 `user`。其他角色在此阶段不需要,脚本化用户轮次不需要它们,且会使恢复语义模糊。 +- `input.turns[*].content` 不得为空。 +- `post_condition.on_fail` 必须为空、`fail` 或 `skip_remaining`;空表示 `fail`。 +- post condition 必须包含 `must_contain_any`、`must_contain_all` 或 `must_not_contain` 中的至少一个。 +- `capture[*].variable` 必须匹配保守的标识符模式,如 `^[A-Za-z_][A-Za-z0-9_]*$`。 +- 每个 capture 规则必须指定恰好一个提取器:`pattern` 或 `jsonpath`。 +- 正则表达式必须在验证时编译成功。 +- `timeout_seconds` 必须为非负数。 +- 每轮 judge 规则必须引用轮次号 >= 1,且当总轮次已知时 <= `len(input.turns)`。 +- `turn_response_contains` 必须指定 `contains_all` 或 `contains_any`。 +- `turn_response_not_contains.not_contains` 是必需的。 +- `tool_called_in_turn.name` 和 `tool_not_called_in_turn.name` 是必需的。 + +设计约束:验证器不得了解 agent 实现或运行时行为。 + +### 2. Agent Resume 边界 + +文件: + +- `internal/agent/agent.go` +- `internal/agent/claude_code.go` +- `internal/agent/qodercli.go` +- `internal/agent/codex.go` +- `internal/agent/*_test.go` + +添加一个小可选接口,不改变 `Agent`: + +```go +type SessionResumer interface { + RunTurn(ctx context.Context, rt Runtime, opts ExecOptions, message transcript.Message, sessionID string) (*SessionResult, error) +} +``` + +添加 `SessionResult.SessionID string`,使 evaluator 可以在无需类型特定解析的情况下恢复会话。 + +职责: + +- Agent 适配器知道如何启动/恢复它们自己的 CLI 会话。 +- Agent 适配器将会话 ID 规范化为 `SessionResult.SessionID`。 +- Evaluator 仅检查 `SessionResumer` 并将 ID 向前传递。 + +实现顺序: + +1. `claude_code`:第一轮已生成 UUID 并传递 `--session-id`。即使输出解析未返回该 ID,也从生成的 ID 填充 `SessionResult.SessionID`。使用 `claude --resume -p` 添加 `RunTurn`。 +2. `qodercli`:切换或添加 JSON 输出路径以解析 `sessionId`。使用精确的 `-r ` 添加 `RunTurn`,而非"继续最新会话"。 +3. `codex`:仅在解决安全的会话关联后实现。优先使用 CLI 输出的会话 ID。如果不可用,使用用例隔离的会话目录或会话目录的锁定前后 diff。不要在并发 `cases.parallelism` 下使用全局"最新会话"查找。 +4. `custom`:暂不实现 `SessionResumer`。自定义引擎已在一个请求中接受完整消息历史,因此它们保持回退能力,直到未来设计自定义恢复契约。 + +边界情况: + +- 第一轮后 `sessionID` 为空但仍有更多轮次时,成为明确的用例执行错误。 +- 恢复命令的认证/限流处理应重用现有首次运行信号检测。 +- `ExecOptions.ArtifactDir`、超时、环境变量、模型、工作空间和可观测性元数据必须在首次和恢复轮次中一致遵守。 + +### 3. Evaluator 多轮引擎 + +文件: + +- `internal/evaluator/evaluator.go` +- 可能需要新的专注辅助文件,例如 `internal/evaluator/multiturn.go` +- `internal/evaluator/evaluator_test.go` + +添加 evaluator 拥有的类型: + +```go +type TurnStatus string + +const ( + TurnCompleted TurnStatus = "completed" + TurnSkipped TurnStatus = "skipped" + TurnFailed TurnStatus = "failed" + TurnError TurnStatus = "error" +) + +type TurnResult struct { + TurnNumber int + Content string + Response string + Transcript transcript.Transcript + SessionResult *agent.SessionResult + Status TurnStatus + Reason string + CapturedVars map[string]string +} +``` + +执行行为: + +- 仅当 `len(input.turns) > 1` 时分支到多轮。 +- 对 `input.prompt` 和单轮 `input.turns` 保持现有单轮路径不变。 +- 为每个用例准备一次运行时、skills、MCP、workspace diff hooks、artifact 目录和 judge 配置,然后在该运行时中执行所有轮次。 +- 第 1 轮调用 `runAgent.Run`,传入单个用户消息。 +- 第 2..N 轮调用 `resumer.RunTurn`。 +- 每轮可能有更窄的 `turn.timeout_seconds`;整体用例超时仍限制整个用例。 +- 每轮后评估 `post_condition`。 +- 如果 `post_condition` 失败且 `on_fail: fail`,标记用例 `FAIL` 且不执行后续轮次。 +- 如果失败且 `on_fail: skip_remaining`,标记剩余轮次跳过,仅当场景未产生可 judge 的完成路径时返回 `SKIP`。如果有完成的轮次加上跳过的后续轮次,当配置了 judge 时仍应运行,每轮规则决定最终状态。 +- 在成功的 post condition 后捕获变量。 +- 模板替换在发送轮次之前立即进行。未解析的 `{{variable}}` 占位符为 `ERROR`,而非发送给 agent 的原始提示文本。 + +结果聚合: + +- 在最终 transcript 中保留每个用户轮次和助手响应,并带有正确的轮次号。 +- 优先使用适配器返回的每轮 transcript;当适配器返回累积 transcript 时,在追加前进行规范化/去重。 +- 最终消息是最后一个完成的轮次响应。 +- `Turns` 是完成的 agent 轮次数。 +- 当适配器返回每轮用量时,token 计数跨轮次求和。 +- 错误/超时的 artifact 处理必须与当前单轮行为匹配。 +- `Expect` 和现有全局 judge 规则针对最终聚合会话进行评估。 +- 每轮 judge 规则接收 `judge.Input.TurnResults`。 + +避免递归陷阱:回退不得以无限重新进入多轮分支的方式调用 `executeCaseOnce`。通过专用的单次辅助函数或布尔执行模式守卫实现回退。 + +### 4. Post Conditions 和 Capture + +文件: + +- `internal/evaluator/multiturn.go` +- `internal/evaluator/multiturn_test.go` + +Post-condition 语义: + +- `must_contain_all`:所有必需字符串必须出现。 +- `must_contain_any`:至少一个字符串必须出现。 +- `must_not_contain`:不得出现任何字符串。 +- 匹配最初遵循现有 `output_contains` 行为:默认区分大小写。如果需要不区分大小写的行为,在后续 schema 扩展中明确指定,而非静默偏离。 +- 空 `on_fail` 表示 `fail`。 +- 失败原因必须命名缺失或禁止的关键词,以便调试。 + +Capture 语义: + +- 正则表达式捕获支持命名组 `(?P...)`;如果不存在,允许恰好一个捕获组作为便利。 +- 无匹配、无效组或空值为执行 `ERROR`。 +- JSONPath 捕获为第 3 阶段。如果避免新依赖很重要,第 1/2 阶段保持仅正则表达式。 +- 捕获的值存储在 evaluator 本地映射中,仅作用域于一个用例执行。 +- 变量从不跨用例、基线变体、重试或迭代共享。 +- 如果后续轮次引用未知变量,在调用 agent 之前失败。 + +### 5. Judge 每轮断言 + +文件: + +- `internal/judge/judge.go` +- `internal/judge/rule_based.go` +- `internal/judge/rule_based_test.go` + +添加 judge 可见的轮次结果: + +```go +type TurnResult struct { + TurnNumber int `json:"turn_number"` + Content string `json:"content"` + Response string `json:"response"` + Transcript transcript.Transcript `json:"transcript"` + Status string `json:"status"` + Reason string `json:"reason,omitempty"` +} +``` + +添加 `TurnResults []TurnResult` 到 `judge.Input`。 + +规则: + +- `turn_response_contains`:检查一个完成的轮次响应,使用 `contains_all` 和/或 `contains_any`。 +- `turn_response_not_contains`:检查禁止字符串不在一个完成的轮次响应中。 +- `tool_called_in_turn`:检查该轮次 transcript 中的工具调用,使用与 `tool_called` 相同的部分参数匹配语义。 +- `tool_not_called_in_turn`:检查该轮次 transcript 中不存在。 + +失败行为: + +- 缺失轮次对正检查是失败的断言。 +- 缺失轮次在此实现中也应对 `tool_not_called_in_turn` 失败,因为验证应已拒绝不可能的轮次号,且缺失的执行轮次通常意味着场景未按预期运行。 +- 跳过、失败或出错的轮次不是"完成的";针对它的响应断言失败,证据中命名状态。 + +### 6. 报告和 Artifacts + +文件: + +- `internal/report/reporter.go` +- `internal/report/html.go` +- `internal/report/junit.go`(如果 JUnit 应在失败文本中包含轮次详情) +- `internal/runner/runner.go` +- 报告测试 + +添加 `TurnResults` 到 `evaluator.EvalResult` 和 `report.CaseResult`。 + +报告行为: + +- `result.json` 和 `report.json` 包含 `turn_results`。 +- `response.md` 保持为最终响应以兼容。 +- 当 `report.artifacts` 包含 `transcript` 时,添加轮次 transcript artifact,尽可能使用现有 artifact 流。 +- HTML 应在紧凑的用例详情部分显示每轮提示、响应、状态和原因。 +- JUnit 可保持用例级别,但断言证据应包含轮次号,使 CI 失败可操作。 + +兼容性规则:现有报告字段保持其含义。新字段为可选,单轮用例中省略。 + +### 7. 文档和示例 + +文件: + +- `docs/guide/writing-evals.md` +- `docs/zh/guide/writing-evals.md` +- `skills/skill-upper/references/case-yaml.md` +- `skills/skill-upper/assets/case.yaml.tmpl` +- `e2e/testdata/multi-turn-conversation/*` +- `CHANGELOG.md`(当实现用户可见时) + +文档应解释: + +- 何时使用 `input.prompt` 与 `input.turns`。 +- `post_condition` 作为轮次间门控,而非 judge 的替代品。 +- Capture/模板语法和失败行为。 +- Agent 支持矩阵和回退行为。 +- 每轮断言示例,用于阶段门控、确认、澄清、迭代优化和跨轮次引用。 + +更新现有 `e2e/testdata/multi-turn-conversation` 用例,使其使用真实的 `input.turns`,而非描述假轮次的单个提示。 + +## 分阶段交付 + +### 第 0 阶段:安全测试框架 + +目标:在行为变更之前准备测试和接口。 + +- 添加单元测试记录当前单轮兼容性。 +- 在 evaluator 测试中添加 fake `SessionResumer` agent。 +- 添加报告 fixture 期望,用于可选的 `turn_results`。 + +验证: + +- `go test -race ./internal/config ./internal/evaluator ./internal/judge ./internal/report` + +### 第 1 阶段:真实顺序多轮核心 + +目标:为支持恢复的 agent 顺序运行脚本化的 `input.turns`。 + +- 添加 `SessionResult.SessionID`。 +- 添加 `SessionResumer`。 +- 首先实现 `claude_code` 恢复。 +- 添加 evaluator 多轮分支、轮次循环、post conditions、聚合。 +- 添加带明确警告的回退模式,无递归分支。 +- 添加 JSON 结果 `turn_results`。 + +验证: + +- 单元测试覆盖 happy path、`post_condition fail`、`post_condition skip_remaining`、空 session ID、轮次超时和回退。 +- 现有单轮 evaluator 测试不变通过。 + +### 第 2 阶段:每轮 Judge 断言 + +目标:让 rule-based judge 断言特定轮次响应和工具调用。 + +- 添加每轮规则结构和验证。 +- 添加 `judge.Input.TurnResults`。 +- 实现四个每轮规则评估器。 +- 添加测试覆盖越界、跳过轮次、失败轮次、正负匹配和工具参数。 + +验证: + +- `go test -race ./internal/config ./internal/judge ./internal/evaluator` + +### 第 3 阶段:Capture 和模板变量 + +目标:支持确定性的跨轮次值传递。 + +- 添加正则表达式捕获。 +- 添加模板渲染和未解析变量检测。 +- 决定是否现在包含 JSONPath。如果是,通过 `go get` 添加依赖并记录许可证;如果否,保持 `jsonpath` 验证被拒绝,直到后续版本。 +- 添加测试覆盖正则命名组、一个未命名组、无匹配、空捕获、无效变量和未解析占位符。 + +验证: + +- `go test -race ./internal/config ./internal/evaluator` + +### 第 4 阶段:更广泛的 Agent 支持和 E2E + +目标:使内置 agent 支持实用且被覆盖。 + +- 实现 `qodercli` 精确会话恢复。 +- 仅在有安全会话关联时实现 `codex`。如果不安全,对 codex 返回明确的不支持多轮错误,而非冒险跨用例污染。 +- 将 `e2e/testdata/multi-turn-conversation` 转换为真实的多轮用例。 +- 添加或更新 E2E 测试,在可能时使用 mock/custom agent 支持运行,并保持真实 agent E2E 像现有引擎测试一样受保护。 + +验证: + +- `make test` +- `make verify` +- 如果触及 `e2e/` 或 `internal/runner/` 行为,也运行 `make e2e`。 + +## 测试矩阵 + +单元测试: + +- Config 解析所有新字段。 +- Config 拒绝歧义的 prompt+turns。 +- Config 拒绝无效角色、空内容、无效 post conditions、无效 capture 规则、无效正则、无效超时和坏的每轮 judge 规则。 +- Evaluator 按顺序执行轮次并将相同的会话 ID 传递给恢复的轮次。 +- Evaluator 在 `post_condition fail` 时停止。 +- Evaluator 在 `skip_remaining` 时跳过剩余轮次。 +- Evaluator 处理第一轮后缺失的会话 ID。 +- Evaluator 在用例超时内尊重每轮超时。 +- Evaluator 聚合 transcript、tokens、turns、最终消息和错误。 +- Capture/模板渲染成功和失败,带有清晰的诊断信息。 +- Judge 每轮响应断言正确通过/失败。 +- Judge 每轮工具断言正确通过/失败。 +- 报告对单轮用例省略 `turn_results`,对多轮用例包含它们。 + +集成测试: + +- 带 fake resumer 的多轮用例通过完整的 evaluator + judge。 +- 多轮回退产生警告/结果模式,每轮断言明确失败。 +- 基线模式为 `with_skill` 和 `without_skill` 保持独立会话。 +- 重试使用新的运行时/会话和空的捕获变量重新运行整个用例。 +- 即使后续轮次出错,workspace artifacts 也被收集。 + +E2E 测试: + +- 阶段门控跳过尝试:第 1 轮进入阶段,第 2 轮尝试跳过,judge 断言第 2 轮拒绝。 +- 双重确认:第 1 轮请求确认,第 2 轮仅在确认后执行。 +- 捕获/引用:第 1 轮输出标识符,第 2 轮通过 `{{variable}}` 引用它。 +- 报告验证:`result.json`、`report.json` 和 HTML 包含轮次详情和完整 transcript。 + +手动验证: + +- 使用 `cases.parallelism: 1` 运行真实的 `claude_code` 多轮用例。 +- 使用 `cases.parallelism: 2` 运行,验证会话不交叉。 +- 运行不支持的 agent/自定义引擎,验证回退或明确的不支持诊断。 + +声明实现完成前的最终检查门: + +```bash +make fmt +make verify +make test +make e2e # 如果包含 e2e/ 或 internal/runner/ 变更则必需 +``` + +## 边界情况和决策 + +- `input.prompt` 加 `input.turns`:验证时拒绝。 +- 单个 `input.turns` 项:使用现有单轮路径,除非存在 post-condition/capture。如果存在 post-condition/capture,使用多轮机制,使这些字段生效。 +- 不支持的恢复:仅当用例未声明 post conditions、capture 或每轮断言时,默认使用显式回退。否则返回 `ERROR`,因为回退无法满足声明的语义。 +- 恢复 agent 的累积 transcripts:按轮次号规范化,避免在聚合 transcript 中重复先前的消息。 +- 无轮次号的工具调用:当它们是每轮 `RunTurn` 结果的一部分时,分配当前轮次。 +- Agent 返回空响应:post-condition 和 judge 规则正常失败;无错误的空最终响应对于无响应断言的用例仍然有效。 +- 轮次超时:标记用例 `ERROR`,保留部分轮次结果,收集 artifacts。 +- 用例超时:当前超时注解保持为外部权威。 +- 捕获值包含花括号:视为纯文本;不递归渲染捕获的值。 +- 捕获变量冲突:后续捕获可能覆盖早期变量,但轮次结果记录每轮捕获的内容。如果重复变量在实践中变得令人困惑,考虑仅添加验证器警告。 +- 基线模式:`with_skill` 和 `without_skill` 绝不得共享会话 ID 或捕获变量。 +- 并行用例:绝不要使用"最新会话"API 进行恢复。所有会话关联必须精确。 +- 报告和评分:`post_condition fail` 是用例 `FAIL`;基础设施错误如无效模板、捕获失败、恢复失败和超时是 `ERROR`;由于 `skip_remaining` 导致场景不适用可能是 `SKIP`。 + +## 待讨论问题 + +- 不支持的 agent 应广泛使用单次回退,还是仅当用例未声明 post conditions、capture 或每轮断言时才允许回退? +- 每轮字符串匹配应像现有 `output_contains` 一样区分大小写,还是新的每轮规则应不区分大小写? +- JSONPath 捕获应包含在首次实现中,还是在正则表达式捕获稳定后发布? +- 带 `post_condition` 的单轮 `input.turns` 是否应立即使用多轮机制,即使只有一轮? diff --git a/docs/guide/writing-evals.md b/docs/guide/writing-evals.md index 232c358a..e835f0c4 100644 --- a/docs/guide/writing-evals.md +++ b/docs/guide/writing-evals.md @@ -465,6 +465,125 @@ context: --- +## Multi-turn conversations + +Use `input.turns` instead of `input.prompt` when your evaluation requires +multiple sequential interactions with the agent — for example, iterative +refinement, phase-gated workflows, or clarification loops. + +### When to use `input.prompt` vs `input.turns` + +| Scenario | Use | +|----------|-----| +| Single instruction, judge the final output | `input.prompt` | +| Multi-step workflow with inter-turn gates | `input.turns` | +| Iterative refinement (e.g. "now improve naming") | `input.turns` | +| Agent must reject invalid requests mid-conversation | `input.turns` | + +### Basic multi-turn case + +```yaml +input: + turns: + - role: user + content: "Implement a binary search function in Go." + post_condition: + must_contain_all: ["func", "binary"] + on_fail: fail + - role: user + content: "Add unit tests for the function you wrote." + post_condition: + must_contain_any: ["Test", "t.Run", "testing"] + on_fail: fail +``` + +### `post_condition` — inter-turn gate + +`post_condition` checks the agent's response after each turn. It is a **gate**, +not a replacement for the judge. Use it to ensure the conversation stays on +track before sending the next turn. + +Fields: + +- `must_contain_all`: all strings must appear in the response. +- `must_contain_any`: at least one string must appear. +- `must_not_contain`: none of the strings may appear. +- `on_fail`: what happens when the condition fails: + - `fail` (default): mark the case as FAIL immediately. + - `skip_remaining`: skip all subsequent turns; the case result depends on + what the judge sees. + +### `capture` — template variables + +Capture values from a turn response for use in later turns: + +```yaml +input: + turns: + - role: user + content: "Generate a session token." + capture: + - variable: token + pattern: "token[=: ]+(?P[A-Za-z0-9]+)" + - role: user + content: "Verify the token {{token}} is valid." +``` + +Capture semantics: + +- Named group `(?P...)` is preferred; if absent, exactly one unnamed + capture group is allowed. +- Extractor type: `pattern` (regex) or `jsonpath` — specify exactly one. +- No match or empty value → the case enters ERROR state. +- Variables are scoped to the current case execution only (never shared across + cases, retries, or baseline variants). +- Referencing an unknown variable fails the turn before the agent is invoked. + +### Agent support matrix + +| Engine | Multi-turn support | Mechanism | +|--------|-------------------|------------| +| `claude_code` | Yes | `--resume` flag with session ID | +| `qodercli` | Yes | `-r ` flag | +| `codex` | Yes | `codex resume ` command | +| `qwen_code` | Not yet | Falls back to batch mode | +| `custom` | Not yet | Falls back to batch mode | + +When an agent does not implement session resumption, all turns are concatenated +and sent as a single prompt. A warning is logged. + +### Per-turn judge assertions + +The rule-based judge supports per-turn assertions: + +```yaml +judge: + type: rule_based + success: + - turn_response_contains: + turn: 1 + contains_all: ["binary_search"] + - turn_response_not_contains: + turn: 2 + not_contains: ["TODO", "FIXME"] + - tool_called_in_turn: + turn: 1 + name: write_file + args: + path: "search.go" + - tool_not_called_in_turn: + turn: 2 + name: delete_file +``` + +Failure behavior: + +- Missing turn (not executed) → assertion fails. +- Skipped/failed/errored turn → assertion fails (only "completed" turns are + assertable). + +--- + ## Grading strategies Grading happens in two layers: **expect** (gating checks) and **judge** (quality assessment). @@ -540,6 +659,9 @@ Let an LLM grade against rubric criteria — useful when semantic understanding judge: type: agent_judge model: anthropic/claude-sonnet-4-6 # Model used by the judge + skills: # Optional: judge-only Skills + - source: local_path + path: evals/fixtures/judge-rubric criteria: # Natural-language rubric - "Identifies a real bug with an accurate location" - "Does not flag correct code as a bug" @@ -548,6 +670,14 @@ judge: timeout_seconds: 60 # Optional: bound a single judge call (0 = no judge-level deadline, parent case timeout still applies) ``` +`judge.skills` is supported only for `agent_judge`. These Skills are installed +into the judge agent, not the run agent, and top-level `skills` are not +automatically installed into the judge. In benchmark mode, judge Skills are +installed for both `with_skill` and `without_skill` runs because they are +grading tooling, not the Skill under test. Installation uses each Agent +adapter's native Skill mechanism; skill-up does not concatenate Skill files +into the judge prompt. + > **Cost note:** `agent_judge` consumes additional tokens. Prefer `expect` or `rule_based` for deterministic checks and reserve `agent_judge` for assertions that genuinely require semantic understanding. --- diff --git a/docs/zh/guide/writing-evals.md b/docs/zh/guide/writing-evals.md index 25663ea6..6bb5d78c 100644 --- a/docs/zh/guide/writing-evals.md +++ b/docs/zh/guide/writing-evals.md @@ -447,6 +447,118 @@ context: --- +## 多轮对话 + +当评测需要多次顺序交互时(例如迭代优化、阶段门控工作流、澄清循环),使用 +`input.turns` 代替 `input.prompt`。 + +### 何时使用 `input.prompt` vs `input.turns` + +| 场景 | 使用 | +|------|------| +| 单条指令,评判最终输出 | `input.prompt` | +| 多步骤工作流,含轮间门控 | `input.turns` | +| 迭代优化(如"改善命名") | `input.turns` | +| Agent 需在对话中拒绝无效请求 | `input.turns` | + +### 基本多轮用例 + +```yaml +input: + turns: + - role: user + content: "用 Go 实现一个二分查找函数。" + post_condition: + must_contain_all: ["func", "binary"] + on_fail: fail + - role: user + content: "为刚才写的函数添加单元测试。" + post_condition: + must_contain_any: ["Test", "t.Run", "testing"] + on_fail: fail +``` + +### `post_condition` — 轮间门控 + +`post_condition` 在每轮结束后检查 Agent 的响应。它是**门控**而非 judge 的替代品, +用于确保对话在发送下一轮之前保持正轨。 + +字段说明: + +- `must_contain_all`:所有字符串必须出现在响应中。 +- `must_contain_any`:至少一个字符串必须出现。 +- `must_not_contain`:所列字符串均不能出现。 +- `on_fail`:条件失败时的行为: + - `fail`(默认):立即将用例标记为 FAIL。 + - `skip_remaining`:跳过后续所有轮次;最终结果取决于 judge 评判。 + +### `capture` — 模板变量 + +从轮次响应中捕获值,在后续轮次中使用: + +```yaml +input: + turns: + - role: user + content: "生成一个会话令牌。" + capture: + - variable: token + pattern: "token[=: ]+(?P[A-Za-z0-9]+)" + - role: user + content: "验证令牌 {{token}} 是否有效。" +``` + +捕获语义: + +- 优先使用命名分组 `(?P...)`;若无命名组,则允许恰好一个匿名捕获组。 +- 提取器类型:`pattern`(正则)或 `jsonpath` — 必须指定且仅指定一个。 +- 未匹配或空值 → 用例进入 ERROR 状态。 +- 变量作用域仅限当前用例执行(不跨用例、重试或基线变体共享)。 +- 引用未知变量会在调用 Agent 之前使轮次失败。 + +### Agent 支持矩阵 + +| 引擎 | 多轮支持 | 机制 | +|------|---------|------| +| `claude_code` | 是 | `--resume` 标志 + 会话 ID | +| `qodercli` | 是 | `-r ` 标志 | +| `codex` | 是 | `codex resume ` 命令 | +| `qwen_code` | 尚不支持 | 回退为批量模式 | +| `custom` | 尚不支持 | 回退为批量模式 | + +当 Agent 不支持会话恢复时,所有轮次内容会拼接为单条 prompt 发送,并记录警告日志。 + +### 按轮次 Judge 断言 + +规则型 judge 支持按轮次断言: + +```yaml +judge: + type: rule_based + success: + - turn_response_contains: + turn: 1 + contains_all: ["binary_search"] + - turn_response_not_contains: + turn: 2 + not_contains: ["TODO", "FIXME"] + - tool_called_in_turn: + turn: 1 + name: write_file + args: + path: "search.go" + - tool_not_called_in_turn: + turn: 2 + name: delete_file +``` + +失败行为: + +- 不存在的轮次(未执行) → 断言失败。 +- 被跳过/失败/出错的轮次 → 断言失败(仅 "completed" 状态的轮次可被断言)。 + +--- + ## 评估策略 skill-up 的评估分为两层:**expect**(门槛检查)和 **judge**(质量评估)。 @@ -522,6 +634,9 @@ judge: judge: type: agent_judge model: anthropic/claude-sonnet-4-6 # 评审使用的模型 + skills: # 可选:仅供 judge 使用的 Skills + - source: local_path + path: evals/fixtures/judge-rubric criteria: # 评估标准(自然语言描述) - "输出中识别了真实存在的 bug,并给出了准确位置" - "没有将正确代码误报为 bug" @@ -530,6 +645,12 @@ judge: timeout_seconds: 60 # 可选:限制单次 judge 调用时长(0 = 不加 judge 级 deadline,仍受 case timeout 约束) ``` +`judge.skills` 仅支持 `agent_judge`。这些 Skills 会安装到 judge agent, +不会安装到主运行 agent;顶层 `skills` 也不会自动安装到 judge。开启 +benchmark 时,`with_skill` 和 `without_skill` 都会安装 judge Skills,因为它们是 +评分工具,不是被测 Skill。安装过程使用各 Agent adapter 原生的 Skill 机制; +skill-up 不会把 Skill 文件内容拼接进 judge prompt。 + > **成本提示**:`agent_judge` 会消耗额外的 token。建议对关键断言先用 `expect` 或 `rule_based` 做确定性检查,只对需要语义理解的部分使用 `agent_judge`。 --- diff --git a/e2e/testdata/multi-turn-conversation/evals/cases/multi-turn-refinement.yaml b/e2e/testdata/multi-turn-conversation/evals/cases/multi-turn-refinement.yaml index c73c5ead..178c0315 100644 --- a/e2e/testdata/multi-turn-conversation/evals/cases/multi-turn-refinement.yaml +++ b/e2e/testdata/multi-turn-conversation/evals/cases/multi-turn-refinement.yaml @@ -2,17 +2,31 @@ id: multi-turn-refinement title: Iterative refinement - non-recursive quicksort description: > Verify that the Agent can generate a non-recursive quicksort implementation - using a stack. + using a stack, then refine it when asked for better variable naming. tag: functional_test input: - prompt: | - Write a quicksort function in Python. - - Requirements: - 1. Use a stack to implement a non-recursive version - 2. Do not use recursion - - Provide a direct code implementation. + turns: + - role: user + content: | + Write a quicksort function in Python. + Requirements: + 1. Use a stack to implement a non-recursive version + 2. Do not use recursion + Provide a direct code implementation. + post_condition: + must_contain_all: ["stack"] + must_not_contain: ["return quicksort("] + on_fail: fail + capture: + - variable: impl_snippet + pattern: "(def \\w+)" + - role: user + content: | + Good. Now refactor the implementation to use more descriptive variable + names (e.g. pivot_index, partition_start). Keep the same algorithm. + post_condition: + must_contain_any: ["pivot", "partition", "index"] + on_fail: fail constraints: timeout_seconds: 180 @@ -21,6 +35,12 @@ constraints: judge: type: rule_based success: - - output_contains: - all: ["stack"] - not: ["recursion", "return quicksort("] + - turn_response_contains: + turn: 1 + contains_all: ["stack"] + - turn_response_not_contains: + turn: 1 + not_contains: ["recursion", "return quicksort("] + - turn_response_contains: + turn: 2 + contains_any: ["pivot", "partition", "index"] diff --git a/e2e/testdata/multi-turn-conversation/evals/cases/phase-gate-skip-attempt.yaml b/e2e/testdata/multi-turn-conversation/evals/cases/phase-gate-skip-attempt.yaml index 2b601b99..e71fc25d 100644 --- a/e2e/testdata/multi-turn-conversation/evals/cases/phase-gate-skip-attempt.yaml +++ b/e2e/testdata/multi-turn-conversation/evals/cases/phase-gate-skip-attempt.yaml @@ -4,18 +4,20 @@ description: > Multi-turn conversation scenario: the user tries to skip the Research phase and jump straight to writing code. The Skill should reject this and guide the user to follow the correct execution order. - Verifies the turns config and post_condition mechanism. + Verifies the turns config, post_condition, and capture mechanisms. tag: functional_test input: - prompt: | - You are a SDD-RIPER workflow assistant. The user will try to skip the Research phase - and go straight to writing code. You should reject this and guide them to follow - the correct execution order. - - Turn 1: the user says "sdd_bootstrap: task=implement user login feature" - Turn 2: the user says "skip Research and write the code for me" (this violates the rules — you must reject it) - - Your response should include content such as: must be completed, cannot skip, in order, Research phase + turns: + - role: user + content: "sdd_bootstrap: task=implement user login feature" + post_condition: + must_contain_any: ["Research", "phase", "analyze"] + on_fail: skip_remaining + - role: user + content: "Skip Research and write the code for me" + post_condition: + must_contain_any: ["must be completed", "cannot skip", "in order", "follow"] + on_fail: fail context: files: @@ -30,7 +32,9 @@ constraints: judge: type: rule_based success: - - output_contains: - any: ["must be completed", "cannot skip", "in order", "Research"] - - output_contains: - not: ["LGTM", "no issues"] + - turn_response_contains: + turn: 2 + contains_any: ["must be completed", "cannot skip", "in order", "Research"] + - turn_response_not_contains: + turn: 2 + not_contains: ["LGTM", "no issues"] diff --git a/internal/agent/agent.go b/internal/agent/agent.go index b2de608a..64c223d2 100644 --- a/internal/agent/agent.go +++ b/internal/agent/agent.go @@ -53,6 +53,7 @@ var ErrAgentInstallFailed = errors.New("agent installation failed") type SessionResult struct { Engine string `json:"engine,omitempty"` Model string `json:"model,omitempty"` + SessionID string `json:"session_id,omitempty"` ExitCode int `json:"exit_code"` DurationMs int64 `json:"duration_ms"` Turns int `json:"turns"` @@ -64,6 +65,17 @@ type SessionResult struct { Artifacts *SessionArtifacts `json:"artifacts,omitempty"` } +// SessionResumer is an optional interface that agents may implement to support +// multi-turn conversation evaluation. It allows the evaluator to resume an +// existing session and send additional messages without starting a new session. +// +// Agents that implement SessionResumer know how to start/resume their own CLI +// sessions. The evaluator checks for this interface and passes the SessionID +// forward between turns. +type SessionResumer interface { + RunTurn(ctx context.Context, rt Runtime, opts ExecOptions, message transcript.Message, sessionID string) (*SessionResult, error) +} + // SessionArtifacts holds artifacts produced during an agent session. type SessionArtifacts struct { WorkspaceDiff string `json:"workspace_diff,omitempty"` @@ -138,6 +150,13 @@ type Agent interface { CheckCredentials(ctx context.Context) error } +// Compile-time interface satisfaction checks for SessionResumer. +var ( + _ SessionResumer = (*ClaudeCodeAgent)(nil) + _ SessionResumer = (*QoderCLIAgent)(nil) + _ SessionResumer = (*CodexAgent)(nil) +) + // BaseAgent provides common functionality for agents. // Embedded by specific agent implementations. type BaseAgent struct { diff --git a/internal/agent/agent_test.go b/internal/agent/agent_test.go index ca1eb551..76bdb977 100644 --- a/internal/agent/agent_test.go +++ b/internal/agent/agent_test.go @@ -498,3 +498,27 @@ func TestDetectAgentWithInitParams_ForwardsKwargs(t *testing.T) { t.Fatalf("Cfg.Kwargs[future_key] = %q, want x", got) } } + +func TestExtractSessionIDFromPath(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + path string + want string + }{ + {name: "jsonl extension", path: "/home/user/.qoder/projects/ws/abc-123.jsonl", want: "abc-123"}, + {name: "uuid format", path: "/home/user/.claude/projects/ws/550e8400-e29b-41d4-a716-446655440000.jsonl", want: "550e8400-e29b-41d4-a716-446655440000"}, + {name: "no extension", path: "/path/to/session-id", want: "session-id"}, + {name: "bare filename", path: "my-session.jsonl", want: "my-session"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + got := extractSessionIDFromPath(tt.path) + if got != tt.want { + t.Fatalf("extractSessionIDFromPath(%q) = %q, want %q", tt.path, got, tt.want) + } + }) + } +} diff --git a/internal/agent/claude_code.go b/internal/agent/claude_code.go index e3b18e60..d614d9b2 100644 --- a/internal/agent/claude_code.go +++ b/internal/agent/claude_code.go @@ -151,6 +151,9 @@ func (a *ClaudeCodeAgent) Run(ctx context.Context, rt Runtime, opts ExecOptions, result, err := rt.Exec(ctx, cmd, opts) sessionResult := a.buildSessionResult(ctx, rt, opts, instruction, start, result) + if sessionResult != nil { + sessionResult.SessionID = sessionID + } if authMsg, ok := providerAuthFailureSignal(result, sessionResult); ok { if sessionResult != nil && sessionResult.ExitCode == 0 { sessionResult.ExitCode = 1 @@ -289,6 +292,17 @@ func buildClaudePrintCmd(sessionID, model, instruction string) string { return cmd } +// buildClaudeResumeCmd constructs a claude CLI command that resumes an existing +// session identified by sessionID using --resume and sends a new prompt. +func buildClaudeResumeCmd(sessionID, model, instruction string) string { + cmd := "claude --settings " + shellQuote(`{"disableAllHooks":true}`) + " --resume " + sessionID + " -p --permission-mode=bypassPermissions" + if model != "" { + cmd += " --model " + shellQuote(model) + } + cmd += " " + shellQuote(instruction) + return cmd +} + type claudePrintJSONResult struct { Type string `json:"type"` Subtype string `json:"subtype"` @@ -403,6 +417,98 @@ func (a *ClaudeCodeAgent) buildSessionResult(ctx context.Context, rt Runtime, op } } +// RunTurn resumes an existing Claude Code session and sends a single user +// message. If sessionID is empty, it starts a new session (first turn). +// This implements the SessionResumer interface for multi-turn evaluation. +func (a *ClaudeCodeAgent) RunTurn(ctx context.Context, rt Runtime, opts ExecOptions, message transcript.Message, sessionID string) (*SessionResult, error) { + if sessionID == "" { + // First turn — delegate to Run which creates a new session. + return a.Run(ctx, rt, opts, []transcript.Message{message}) + } + + if err := requireBashOnWindowsHost(rt); err != nil { + return nil, fmt.Errorf("%s: %w", a.Name(), err) + } + start := time.Now() + + envVars := a.claudeRunEnvVars() + opts = a.mergeExecOptionsEnv(ctx, opts, envVars, a.buildAgentObservabilityAttrs(map[string]string{ + "claude_code.session_id": sessionID, + })) + ctx = observability.ContextWithConfiguredAgentSpanAttributes(ctx, opts.Env) + + instruction := message.Content + if err := ensureNodeRuntime(ctx, rt, "claude", opts); err != nil { + return &SessionResult{ + Engine: a.Name(), + SessionID: sessionID, + ExitCode: 1, + DurationMs: time.Since(start).Milliseconds(), + Artifacts: &SessionArtifacts{}, + }, err + } + cmd := buildClaudeResumeCmd(sessionID, a.effectiveModelName(ctx), instruction) + + result, err := rt.Exec(ctx, cmd, opts) + sessionResult := a.buildSessionResult(ctx, rt, opts, instruction, start, result) + if sessionResult != nil { + sessionResult.SessionID = sessionID + } + return a.handleClaudeRunResult(sessionResult, result, err, sessionID, start) +} + +// claudeRunEnvVars builds the environment variable map for Claude Code runs. +func (a *ClaudeCodeAgent) claudeRunEnvVars() map[string]string { + envVars := a.credentialEnvVars(credential.EnvAnthropicAPIKey, credential.EnvAnthropicBaseURL) + if a.Cfg.APIKey != "" { + envVars[credential.EnvAnthropicAuthToken] = a.Cfg.APIKey + } + envVars["CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC"] = "1" + envVars["IS_SANDBOX"] = "1" + if observability.TracingEnabled() { + envVars["CLAUDE_CODE_ENABLE_TELEMETRY"] = "1" + envVars["CLAUDE_CODE_ENHANCED_TELEMETRY_BETA"] = "1" + envVars["ENABLE_ENHANCED_TELEMETRY_BETA"] = "1" + } + return envVars +} + +// handleClaudeRunResult checks provider auth/rate-limit signals and execution +// errors, returning the appropriate SessionResult and error. +func (a *ClaudeCodeAgent) handleClaudeRunResult(sessionResult *SessionResult, result ExecResult, err error, sessionID string, start time.Time) (*SessionResult, error) { + if authMsg, ok := providerAuthFailureSignal(result, sessionResult); ok { + if sessionResult != nil && sessionResult.ExitCode == 0 { + sessionResult.ExitCode = 1 + } + return sessionResult, fmt.Errorf("claude-code authentication failed: %s", authMsg) + } + if rateLimitMsg, ok := providerRateLimitSignal(result, sessionResult); ok { + if sessionResult != nil && sessionResult.ExitCode == 0 { + sessionResult.ExitCode = 1 + } + return sessionResult, fmt.Errorf("claude-code provider rate limit: %s", rateLimitMsg) + } + if err != nil { + if sessionResult == nil { + sessionResult = &SessionResult{ + Engine: a.Name(), + SessionID: sessionID, + ExitCode: 1, + DurationMs: time.Since(start).Milliseconds(), + Stderr: result.Stderr, + Artifacts: &SessionArtifacts{}, + } + } + return sessionResult, fmt.Errorf("claude-code resume failed: %w", err) + } + + if result.ExitCode != 0 { + return sessionResult, fmt.Errorf("claude-code resume failed (exit %d): %s", result.ExitCode, result.Stderr) + } + + return sessionResult, nil +} + func buildClaudeTextSessionResult(engine, instruction string, start time.Time, result ExecResult) *SessionResult { sessionResult := &SessionResult{ Engine: engine, diff --git a/internal/agent/claude_code_test.go b/internal/agent/claude_code_test.go index 0eb7a0de..40093033 100644 --- a/internal/agent/claude_code_test.go +++ b/internal/agent/claude_code_test.go @@ -119,6 +119,147 @@ func TestBuildClaudeRunCmd_WithModel(t *testing.T) { } } +func TestBuildClaudeResumeCmd(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + sessionID string + model string + instruction string + wantParts []string + wantAbsent []string + }{ + { + name: "with model", + sessionID: "abc-123", + model: "claude-sonnet-4-6", + instruction: "continue", + wantParts: []string{"--resume abc-123", "-p", "--model 'claude-sonnet-4-6'", "'continue'"}, + wantAbsent: []string{"--session-id"}, + }, + { + name: "without model", + sessionID: "def-456", + model: "", + instruction: "next step", + wantParts: []string{"--resume def-456", "-p", "'next step'"}, + wantAbsent: []string{"--model", "--session-id"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + cmd := buildClaudeResumeCmd(tt.sessionID, tt.model, tt.instruction) + for _, part := range tt.wantParts { + if !strings.Contains(cmd, part) { + t.Fatalf("expected command to contain %q, got %q", part, cmd) + } + } + for _, absent := range tt.wantAbsent { + if strings.Contains(cmd, absent) { + t.Fatalf("expected command to NOT contain %q, got %q", absent, cmd) + } + } + }) + } +} + +func TestClaudeCodeRun_PopulatesSessionID(t *testing.T) { + t.Parallel() + + rt := &claudeCodeTestRuntime{ + workspace: t.TempDir(), + execResult: runtime.ExecResult{ + Stdout: "OK\n", + ExitCode: 0, + }, + } + ag := NewClaudeCodeAgent(Config{}) + + result, err := ag.Run(context.Background(), rt, ExecOptions{}, []transcript.Message{{ + Role: transcript.RoleUser, + Content: "Reply OK.", + Turn: 1, + }}) + if err != nil { + t.Fatalf("run claude-code: %v", err) + } + if result.SessionID == "" { + t.Fatal("expected SessionID to be populated after Run") + } + // Verify the session ID is a valid UUID-like format (contains hyphens) + if !strings.Contains(result.SessionID, "-") { + t.Fatalf("expected SessionID to look like a UUID, got %q", result.SessionID) + } +} + +func TestClaudeCodeRunTurn_FirstTurnDelegatesToRun(t *testing.T) { + t.Parallel() + + rt := &claudeCodeTestRuntime{ + workspace: t.TempDir(), + execResult: runtime.ExecResult{ + Stdout: "OK\n", + ExitCode: 0, + }, + } + ag := NewClaudeCodeAgent(Config{}) + + result, err := ag.RunTurn(context.Background(), rt, ExecOptions{}, transcript.Message{ + Role: transcript.RoleUser, + Content: "start conversation", + Turn: 1, + }, "") + if err != nil { + t.Fatalf("RunTurn (first turn): %v", err) + } + if result.SessionID == "" { + t.Fatal("expected SessionID after first turn") + } + // First turn uses --session-id (not --resume) + if strings.Contains(rt.agentCommand, "--resume") { + t.Fatalf("first turn should use --session-id, not --resume: %q", rt.agentCommand) + } +} + +func TestClaudeCodeRunTurn_ResumeUsesCorrectFlag(t *testing.T) { + t.Parallel() + + rt := &claudeCodeTestRuntime{ + workspace: t.TempDir(), + execResult: runtime.ExecResult{ + Stdout: "Resumed answer\n", + ExitCode: 0, + }, + } + ag := NewClaudeCodeAgent(Config{ModelName: "claude-sonnet-4-6"}) + + result, err := ag.RunTurn(context.Background(), rt, ExecOptions{}, transcript.Message{ + Role: transcript.RoleUser, + Content: "follow up", + Turn: 2, + }, "existing-session-id") + if err != nil { + t.Fatalf("RunTurn (resume): %v", err) + } + if result.SessionID != "existing-session-id" { + t.Fatalf("expected SessionID = %q, got %q", "existing-session-id", result.SessionID) + } + if !strings.Contains(rt.agentCommand, "--resume existing-session-id") { + t.Fatalf("expected --resume flag, got %q", rt.agentCommand) + } + if strings.Contains(rt.agentCommand, "--session-id") { + t.Fatalf("resume should not use --session-id: %q", rt.agentCommand) + } + if !strings.Contains(rt.agentCommand, "--model 'claude-sonnet-4-6'") { + t.Fatalf("expected model flag in resume command: %q", rt.agentCommand) + } + if !strings.Contains(rt.agentCommand, "'follow up'") { + t.Fatalf("expected instruction in resume command: %q", rt.agentCommand) + } +} + func TestBuildStreamJSON_EncodesStringAsTextBlocks(t *testing.T) { t.Parallel() @@ -708,6 +849,7 @@ type claudeCodeTestRuntime struct { workspace string execResult runtime.ExecResult lastCommand string + agentCommand string // first non-probe, non-intercepted command lastExecEnv map[string]string execCount int probeResponseStdout string // canned stdout for PATH probe; defaults to a fake bin @@ -753,9 +895,15 @@ func (r *claudeCodeTestRuntime) Exec(_ context.Context, command string, opts run if strings.Contains(command, "if command -v 'claude' >/dev/null 2>&1; then exit 0; fi") { return runtime.ExecResult{ExitCode: 0}, nil } + // Session file lookup scripts start with printenv HOME; treat them as + // background operations that do not overwrite the agent command. + if strings.HasPrefix(command, "home=$(printenv HOME)") { + return runtime.ExecResult{}, errors.New("no session file") + } r.lastCommand = command r.execCount++ if r.execCount == 1 { + r.agentCommand = command r.lastExecEnv = mapsClone(opts.Env) return r.execResult, nil } diff --git a/internal/agent/codex.go b/internal/agent/codex.go index 741e053d..0e7d2952 100644 --- a/internal/agent/codex.go +++ b/internal/agent/codex.go @@ -233,6 +233,67 @@ func (a *CodexAgent) CheckCredentials(ctx context.Context) error { return nil } +// RunTurn implements SessionResumer for multi-turn conversation support. +// First turn (sessionID=="") delegates to Run; subsequent turns use +// `codex resume ` to continue the existing session. +func (a *CodexAgent) RunTurn(ctx context.Context, rt Runtime, opts ExecOptions, message transcript.Message, sessionID string) (*SessionResult, error) { + if sessionID == "" { + // First turn — delegate to Run which creates a new session. + return a.Run(ctx, rt, opts, []transcript.Message{message}) + } + + if err := requireBashOnWindowsHost(rt); err != nil { + return nil, fmt.Errorf("%s: %w", a.Name(), err) + } + start := time.Now() + + instruction := message.Content + lastMessagePath := filepath.Join(rt.Workspace(), ".skill-up", "codex-last-message.txt") + + envVars := a.credentialEnvVars(credential.EnvOpenAIAPIKey, credential.EnvOpenAIBaseURL) + opts = a.mergeExecOptionsEnv(ctx, opts, envVars, a.buildAgentObservabilityAttrs(map[string]string{ + "codex.session_id": sessionID, + })) + ctx = observability.ContextWithConfiguredAgentSpanAttributes(ctx, opts.Env) + + if err := ensureNodeRuntime(ctx, rt, codexEngineName, opts); err != nil { + return &SessionResult{ + Engine: a.Name(), + SessionID: sessionID, + ExitCode: 1, + DurationMs: time.Since(start).Milliseconds(), + Artifacts: &SessionArtifacts{}, + }, err + } + cmd := "mkdir -p " + shellQuote(filepath.Dir(lastMessagePath)) + "\n" + + buildCodexResumeCmdWithLastMessage(sessionID, instruction, a.effectiveModelName(ctx), a.runProviderConfig(ctx), lastMessagePath) + + result, err := rt.Exec(ctx, cmd, opts) + sessionResult := a.buildSessionResult(ctx, rt, opts, instruction, start, result, lastMessagePath) + if sessionResult != nil { + sessionResult.SessionID = sessionID + } + if err != nil { + if sessionResult == nil { + sessionResult = &SessionResult{ + Engine: a.Name(), + SessionID: sessionID, + ExitCode: 1, + DurationMs: time.Since(start).Milliseconds(), + Stderr: result.Stderr, + Artifacts: &SessionArtifacts{}, + } + } + return sessionResult, fmt.Errorf("codex resume failed: %w", err) + } + + if result.ExitCode != 0 { + return sessionResult, fmt.Errorf("codex resume failed (exit %d): %s", result.ExitCode, result.Stderr) + } + + return sessionResult, nil +} + // Run executes codex in non-interactive JSON mode and converts events into a transcript. // //nolint:dupl @@ -411,6 +472,7 @@ func (a *CodexAgent) buildSessionResult( return &SessionResult{ Engine: a.Name(), + SessionID: streamParsed.threadID, ExitCode: result.ExitCode, DurationMs: time.Since(start).Milliseconds(), Turns: codexTurns(trans), @@ -470,6 +532,26 @@ func buildCodexRunCmdWithLastMessage(instruction, model string, provider codexPr return cmd } +// buildCodexResumeCmdWithLastMessage constructs a codex CLI command that resumes +// an existing session identified by sessionID and sends a new user message. +// Note: `codex exec resume` does not support --sandbox; the session inherits +// sandbox mode from its initial creation. We use --dangerously-bypass-approvals-and-sandbox +// to ensure non-interactive execution (approvals are skipped). +func buildCodexResumeCmdWithLastMessage(sessionID, instruction, model string, provider codexProviderConfig, lastMessagePath string) string { + cmd := "codex exec resume --json --skip-git-repo-check --dangerously-bypass-approvals-and-sandbox" + cmd += codexProviderFlags(provider) + if model != "" { + cmd += " -m " + shellQuote(model) + } + if lastMessagePath != "" { + cmd += " --output-last-message " + shellQuote(lastMessagePath) + } + cmd += " " + shellQuote(sessionID) + cmd += " " + shellQuote(instruction) + + return cmd +} + func codexProviderFlags(provider codexProviderConfig) string { if provider.Name == "" { return "" diff --git a/internal/agent/codex_test.go b/internal/agent/codex_test.go index 42c2c5e8..85a7cc62 100644 --- a/internal/agent/codex_test.go +++ b/internal/agent/codex_test.go @@ -330,6 +330,106 @@ func TestCodexRunProviderConfig_OpenAIWithoutBaseURLEmitsNothing(t *testing.T) { } } +func TestCodexRunTurn_FirstTurnDelegatesToRun(t *testing.T) { + t.Parallel() + + threadEvent := `{"type":"thread.started","thread_id":"thread-abc"}` + rt := &codexTestRuntime{ + workspace: t.TempDir(), + execResult: runtime.ExecResult{ + Stdout: threadEvent + "\n", + ExitCode: 0, + }, + } + ag := NewCodexAgent(Config{ModelName: "o3"}) + + result, err := ag.RunTurn(context.Background(), rt, ExecOptions{}, transcript.Message{ + Role: transcript.RoleUser, + Content: "start task", + Turn: 1, + }, "") + if err != nil { + t.Fatalf("RunTurn (first turn): %v", err) + } + if result == nil { + t.Fatal("expected non-nil result") + } + // First turn should use "codex exec", not "codex resume" + if containsCommand(rt.commands, "codex resume") { + t.Fatalf("first turn should not use codex resume: %v", rt.commands) + } + if !containsCommand(rt.commands, "codex exec") { + t.Fatalf("first turn should use codex exec: %v", rt.commands) + } + // SessionID should be extracted from thread event + if result.SessionID != "thread-abc" { + t.Fatalf("expected SessionID = %q, got %q", "thread-abc", result.SessionID) + } +} + +func TestCodexRunTurn_ResumeUsesCorrectCommand(t *testing.T) { + t.Parallel() + + rt := &codexTestRuntime{ + workspace: t.TempDir(), + execResult: runtime.ExecResult{ + Stdout: `{"type":"turn.started"}` + "\n" + `{"type":"item.completed","item":{"type":"agent_message","text":"resumed"}}` + "\n", + ExitCode: 0, + }, + } + ag := NewCodexAgent(Config{ModelName: "o3"}) + + result, err := ag.RunTurn(context.Background(), rt, ExecOptions{}, transcript.Message{ + Role: transcript.RoleUser, + Content: "continue working", + Turn: 2, + }, "thread-xyz-456") + if err != nil { + t.Fatalf("RunTurn (resume): %v", err) + } + if result.SessionID != "thread-xyz-456" { + t.Fatalf("expected SessionID = %q, got %q", "thread-xyz-456", result.SessionID) + } + // Should use "codex exec resume" with the session ID + if !containsCommand(rt.commands, "codex exec resume") { + t.Fatalf("expected codex exec resume command, got %v", rt.commands) + } + if !containsCommand(rt.commands, "'thread-xyz-456'") { + t.Fatalf("expected session ID in command, got %v", rt.commands) + } + if !containsCommand(rt.commands, "'continue working'") { + t.Fatalf("expected instruction in command, got %v", rt.commands) + } +} + +func TestBuildCodexResumeCmdWithLastMessage(t *testing.T) { + t.Parallel() + + cmd := buildCodexResumeCmdWithLastMessage( + "sess-123", "do something", "o3", + codexProviderConfig{}, + "/tmp/last.txt", + ) + if !strings.HasPrefix(cmd, "codex exec resume --json --skip-git-repo-check") { + t.Fatalf("expected codex exec resume prefix, got %q", cmd) + } + if !strings.Contains(cmd, "--dangerously-bypass-approvals-and-sandbox") { + t.Fatalf("expected bypass flag, got %q", cmd) + } + if !strings.Contains(cmd, "'sess-123'") { + t.Fatalf("expected session ID, got %q", cmd) + } + if !strings.Contains(cmd, "'do something'") { + t.Fatalf("expected instruction, got %q", cmd) + } + if !strings.Contains(cmd, "-m 'o3'") { + t.Fatalf("expected model flag, got %q", cmd) + } + if !strings.Contains(cmd, "--output-last-message") { + t.Fatalf("expected --output-last-message flag, got %q", cmd) + } +} + func TestShellQuoteEscapesSingleQuote(t *testing.T) { t.Parallel() diff --git a/internal/agent/qodercli.go b/internal/agent/qodercli.go index 4cfd4a45..4081d44d 100644 --- a/internal/agent/qodercli.go +++ b/internal/agent/qodercli.go @@ -129,10 +129,23 @@ func buildQoderRunCmd(instruction, model string) string { return cmd } +// buildQoderResumeCmd constructs a qodercli command that resumes an existing +// session identified by sessionID and sends a new user prompt. +func buildQoderResumeCmd(instruction, model, sessionID string) string { + cmd := "qodercli --permission-mode=bypass_permissions" + if model != "" { + cmd += " --model " + shellQuote(model) + } + cmd += " -r " + shellQuote(sessionID) + cmd += " -p " + shellQuote(instruction) + return cmd +} + func (a *QoderCLIAgent) buildSessionResult(ctx context.Context, rt Runtime, opts ExecOptions, instruction string, start time.Time, result ExecResult) *SessionResult { var trans transcript.Transcript var finalMsg string var inputTokens, outputTokens int + var sessionID string cleanupCtx, cleanupCancel := sessionCleanupContext(ctx) defer cleanupCancel() @@ -144,9 +157,14 @@ func (a *QoderCLIAgent) buildSessionResult(ctx context.Context, rt Runtime, opts } } + sessionFilePath := findQoderSessionFile(cleanupCtx, rt) + if sessionFilePath != "" { + sessionID = extractSessionIDFromPath(sessionFilePath) + } + var cleanupSession func() generatedFiles, cleanupSession = withDownloadedSession( - cleanupCtx, rt, opts.ArtifactDir, findQoderSessionFile(cleanupCtx, rt), generatedFiles, + cleanupCtx, rt, opts.ArtifactDir, sessionFilePath, generatedFiles, func(artifactPath string) { t, f, inTok, outTok := parseSessionFile(artifactPath) if len(t) > 0 { @@ -167,6 +185,7 @@ func (a *QoderCLIAgent) buildSessionResult(ctx context.Context, rt Runtime, opts } return &SessionResult{ Engine: a.Name(), + SessionID: sessionID, ExitCode: result.ExitCode, DurationMs: time.Since(start).Milliseconds(), Turns: countTurns(trans), @@ -181,6 +200,53 @@ func (a *QoderCLIAgent) buildSessionResult(ctx context.Context, rt Runtime, opts } } +// RunTurn resumes an existing QoderCLI session and sends a single user +// message. If sessionID is empty, it starts a new session (first turn). +// This implements the SessionResumer interface for multi-turn evaluation. +func (a *QoderCLIAgent) RunTurn(ctx context.Context, rt Runtime, opts ExecOptions, message transcript.Message, sessionID string) (*SessionResult, error) { + if sessionID == "" { + // First turn — delegate to Run which creates a new session. + return a.Run(ctx, rt, opts, []transcript.Message{message}) + } + + if err := requireBashOnWindowsHost(rt); err != nil { + return nil, fmt.Errorf("%s: %w", a.Name(), err) + } + start := time.Now() + + instruction := message.Content + cmd := buildQoderResumeCmd(instruction, a.effectiveModelName(ctx), sessionID) + + envVars := a.credentialEnvVars("", "") + opts = a.mergeExecOptionsEnv(ctx, opts, envVars, a.buildAgentObservabilityAttrs(nil)) + ctx = observability.ContextWithConfiguredAgentSpanAttributes(ctx, opts.Env) + + result, err := rt.Exec(ctx, cmd, opts) + sessionResult := a.buildSessionResult(ctx, rt, opts, instruction, start, result) + if sessionResult != nil { + sessionResult.SessionID = sessionID + } + if err != nil { + if sessionResult == nil { + sessionResult = &SessionResult{ + Engine: a.Name(), + SessionID: sessionID, + ExitCode: 1, + DurationMs: time.Since(start).Milliseconds(), + Stderr: result.Stderr, + Artifacts: &SessionArtifacts{}, + } + } + return sessionResult, fmt.Errorf("qodercli resume failed: %w", err) + } + + if result.ExitCode != 0 { + return sessionResult, fmt.Errorf("qodercli resume failed (exit %d): %s", result.ExitCode, result.Stderr) + } + + return sessionResult, nil +} + // findQoderSessionFile resolves the newest matching session JSONL under the Qoder // projects tree for this workspace. Per runtime isolation, HOME and the tree // are read only inside the runtime via Exec (not os.Getenv / host os.Stat). diff --git a/internal/agent/qodercli_test.go b/internal/agent/qodercli_test.go index 6aea7d8f..731c1638 100644 --- a/internal/agent/qodercli_test.go +++ b/internal/agent/qodercli_test.go @@ -203,6 +203,111 @@ func TestBuildQoderRunCmd_WithoutModel(t *testing.T) { } } +func TestBuildQoderResumeCmd(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + instruction string + model string + sessionID string + wantParts []string + wantAbsent []string + }{ + { + name: "with model", + instruction: "continue", + model: modelAuto, + sessionID: "session-abc", + wantParts: []string{"--permission-mode=bypass_permissions", "--model 'auto'", "-r 'session-abc'", "-p 'continue'"}, + }, + { + name: "without model", + instruction: "next", + model: "", + sessionID: "sid-xyz", + wantParts: []string{"-r 'sid-xyz'", "-p 'next'"}, + wantAbsent: []string{"--model"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + cmd := buildQoderResumeCmd(tt.instruction, tt.model, tt.sessionID) + for _, part := range tt.wantParts { + if !strings.Contains(cmd, part) { + t.Fatalf("expected command to contain %q, got %q", part, cmd) + } + } + for _, absent := range tt.wantAbsent { + if strings.Contains(cmd, absent) { + t.Fatalf("expected command to NOT contain %q, got %q", absent, cmd) + } + } + }) + } +} + +func TestQoderCLIRunTurn_FirstTurnDelegatesToRun(t *testing.T) { + t.Parallel() + + rt := &qoderTestRuntime{ + workspace: t.TempDir(), + execResult: runtime.ExecResult{ + Stdout: "hello back\n", + ExitCode: 0, + }, + } + ag := NewQoderCLIAgent(Config{ModelName: modelAuto}) + + result, err := ag.RunTurn(context.Background(), rt, ExecOptions{}, transcript.Message{ + Role: transcript.RoleUser, + Content: "start", + Turn: 1, + }, "") + if err != nil { + t.Fatalf("RunTurn (first turn): %v", err) + } + if result == nil { + t.Fatal("expected non-nil result") + } + // First turn should NOT use -r flag + if strings.Contains(rt.agentCommand, " -r ") { + t.Fatalf("first turn should not use -r flag: %q", rt.agentCommand) + } +} + +func TestQoderCLIRunTurn_ResumeUsesCorrectFlag(t *testing.T) { + t.Parallel() + + rt := &qoderTestRuntime{ + workspace: t.TempDir(), + execResult: runtime.ExecResult{ + Stdout: "resumed answer\n", + ExitCode: 0, + }, + } + ag := NewQoderCLIAgent(Config{ModelName: modelAuto}) + + result, err := ag.RunTurn(context.Background(), rt, ExecOptions{}, transcript.Message{ + Role: transcript.RoleUser, + Content: "follow up", + Turn: 2, + }, "qoder-session-123") + if err != nil { + t.Fatalf("RunTurn (resume): %v", err) + } + if result.SessionID != "qoder-session-123" { + t.Fatalf("expected SessionID = %q, got %q", "qoder-session-123", result.SessionID) + } + if !strings.Contains(rt.agentCommand, "-r 'qoder-session-123'") { + t.Fatalf("expected -r flag with session ID, got %q", rt.agentCommand) + } + if !strings.Contains(rt.agentCommand, "-p 'follow up'") { + t.Fatalf("expected -p flag with instruction, got %q", rt.agentCommand) + } +} + func TestQoderCLIEffectiveModelName_AllowsSupportedModel(t *testing.T) { t.Parallel() @@ -383,6 +488,7 @@ type qoderTestRuntime struct { workspace string execResult runtime.ExecResult lastCommand string + agentCommand string // first non-probe, non-intercepted command lastExecEnv map[string]string probeResponseStdout string mergedEnv map[string]string @@ -412,7 +518,15 @@ func (r *qoderTestRuntime) Exec(_ context.Context, command string, opts runtime. } return runtime.ExecResult{Stdout: stdout}, nil } + // Session file lookup scripts start with printenv HOME; treat them as + // background operations that do not overwrite the agent command. + if strings.HasPrefix(command, "home=$(printenv HOME)") { + return runtime.ExecResult{}, nil + } r.lastCommand = command + if r.agentCommand == "" { + r.agentCommand = command + } if strings.Contains(command, "qodercli --permission-mode=bypass_permissions") || strings.Contains(command, "qodercli -p ") || strings.Contains(command, "qoder.com/install") { diff --git a/internal/agent/session_lookup.go b/internal/agent/session_lookup.go index 1fcaa51d..fc9e30ad 100644 --- a/internal/agent/session_lookup.go +++ b/internal/agent/session_lookup.go @@ -116,6 +116,17 @@ func workspaceKeyForRuntime(rt Runtime) string { return strings.ReplaceAll(workspace, "/", "-") } +// extractSessionIDFromPath extracts a session identifier from a session JSONL +// file path. It strips the directory and .jsonl extension, returning the bare +// filename which agents use as the session identifier for resume. +func extractSessionIDFromPath(sessionPath string) string { + base := filepath.Base(sessionPath) + if ext := filepath.Ext(base); ext != "" { + base = base[:len(base)-len(ext)] + } + return base +} + // buildSessionLookupScript renders the shared shell snippet that picks the // newest *.jsonl under the configured root. func buildSessionLookupScript(lookup agentSessionLookup) string { diff --git a/internal/cli/judge_debug.go b/internal/cli/judge_debug.go index af8eee67..9feccd5a 100644 --- a/internal/cli/judge_debug.go +++ b/internal/cli/judge_debug.go @@ -142,7 +142,7 @@ func runJudgeDebug(cmd *cobra.Command, args []string) error { } var input judgeDebugInput - if err := json.Unmarshal(data, &input); err != nil { + if err := json.Unmarshal(data, &input); err != nil { //nolint:musttag // debug input embeds config structs with existing tags. return fmt.Errorf("parse input JSON: %w", err) } diff --git a/internal/cli/judge_debug_test.go b/internal/cli/judge_debug_test.go index 3c7054b3..81a7f5d7 100644 --- a/internal/cli/judge_debug_test.go +++ b/internal/cli/judge_debug_test.go @@ -25,7 +25,7 @@ import ( // writeJudgeDebugInput serialises a judgeDebugInput to a JSON file and returns the path. func writeJudgeDebugInput(t *testing.T, dir string, input judgeDebugInput) string { t.Helper() - data, err := json.Marshal(input) + data, err := json.Marshal(input) //nolint:musttag // debug input embeds config structs with existing tags. if err != nil { t.Fatalf("marshal judge debug input: %v", err) } diff --git a/internal/config/loader_test.go b/internal/config/loader_test.go index 513125da..a6c4a9a8 100644 --- a/internal/config/loader_test.go +++ b/internal/config/loader_test.go @@ -267,6 +267,54 @@ cases: } } +func TestLoader_LoadEvalConfig_JudgeSkills(t *testing.T) { + t.Parallel() + + tmp := t.TempDir() + evalPath := filepath.Join(tmp, "eval.yaml") + content := `schema_version: v1alpha1 + +environment: + type: none + +engine: + name: claude_code + +cases: + files: + - evals/cases/basic.yaml + +judge: + type: agent_judge + model: test-model + skills: + - source: local_path + path: evals/fixtures/default-judge + - source: local_path + path: evals/fixtures/security-judge + target: ~/.claude/skills/security-judge + criteria: + - criterion one +` + if err := os.WriteFile(evalPath, []byte(content), 0o600); err != nil { + t.Fatalf("failed to write temp eval.yaml: %v", err) + } + + cfg, err := NewLoader(evalPath).LoadEvalConfig() + if err != nil { + t.Fatalf("LoadEvalConfig failed: %v", err) + } + if len(cfg.Judge.Skills) != 2 { + t.Fatalf("Judge.Skills length = %d, want 2", len(cfg.Judge.Skills)) + } + if got := cfg.Judge.Skills[0].Path; got != "evals/fixtures/default-judge" { + t.Fatalf("Judge.Skills[0].Path = %q", got) + } + if got := cfg.Judge.Skills[1].Target; got != "~/.claude/skills/security-judge" { + t.Fatalf("Judge.Skills[1].Target = %q", got) + } +} + // nolint:funlen // table-driven matrix over the documented defaults; keeping // each case inline beats spreading them across helpers. func TestLoader_LoadEvalConfig_AppliesDefaults(t *testing.T) { diff --git a/internal/config/schema.go b/internal/config/schema.go index eec23291..63d37503 100644 --- a/internal/config/schema.go +++ b/internal/config/schema.go @@ -186,10 +186,11 @@ type RetryPolicy struct { // JudgeConfig describes the evaluation strategy. type JudgeConfig struct { - Type string `json:"type" yaml:"type"` // rule_based, script, agent_judge - ScriptPath string `json:"script_path,omitempty" yaml:"script_path,omitempty"` - Model string `json:"model,omitempty" yaml:"model,omitempty"` - Criteria []string `json:"criteria,omitempty" yaml:"criteria,omitempty"` + Type string `json:"type" yaml:"type"` // rule_based, script, agent_judge + ScriptPath string `json:"script_path,omitempty" yaml:"script_path,omitempty"` + Model string `json:"model,omitempty" yaml:"model,omitempty"` + Criteria []string `json:"criteria,omitempty" yaml:"criteria,omitempty"` + Skills []SkillRef `json:"skills,omitempty" yaml:"skills,omitempty"` // PassThreshold is the minimum pass rate for agent_judge. // Nil means "not configured", so the judge layer applies its default of 0.7. // When set explicitly, the value must be in the inclusive range [0.0, 1.0]. @@ -206,11 +207,41 @@ type JudgeConfig struct { // Rule is a single assertion rule for rule_based evaluation. type Rule struct { - OutputContains *OutputContainsRule `json:"output_contains,omitempty" yaml:"output_contains,omitempty"` - ExitCode *int `json:"exit_code,omitempty" yaml:"exit_code,omitempty"` - ToolCalled *ToolCalledRule `json:"tool_called,omitempty" yaml:"tool_called,omitempty"` - FilesExist []string `json:"files_exist,omitempty" yaml:"files_exist,omitempty"` - FilesNotExist []string `json:"files_not_exist,omitempty" yaml:"files_not_exist,omitempty"` + OutputContains *OutputContainsRule `json:"output_contains,omitempty" yaml:"output_contains,omitempty"` + ExitCode *int `json:"exit_code,omitempty" yaml:"exit_code,omitempty"` + ToolCalled *ToolCalledRule `json:"tool_called,omitempty" yaml:"tool_called,omitempty"` + FilesExist []string `json:"files_exist,omitempty" yaml:"files_exist,omitempty"` + FilesNotExist []string `json:"files_not_exist,omitempty" yaml:"files_not_exist,omitempty"` + TurnResponseContains *TurnResponseContainsRule `json:"turn_response_contains,omitempty" yaml:"turn_response_contains,omitempty"` + TurnResponseNotContains *TurnResponseNotContainsRule `json:"turn_response_not_contains,omitempty" yaml:"turn_response_not_contains,omitempty"` + ToolCalledInTurn *ToolCalledInTurnRule `json:"tool_called_in_turn,omitempty" yaml:"tool_called_in_turn,omitempty"` + ToolNotCalledInTurn *ToolNotCalledInTurnRule `json:"tool_not_called_in_turn,omitempty" yaml:"tool_not_called_in_turn,omitempty"` +} + +// TurnResponseContainsRule checks that a specific turn response contains required text. +type TurnResponseContainsRule struct { + Turn int `json:"turn" yaml:"turn"` + ContainsAll []string `json:"contains_all,omitempty" yaml:"contains_all,omitempty"` + ContainsAny []string `json:"contains_any,omitempty" yaml:"contains_any,omitempty"` +} + +// TurnResponseNotContainsRule checks that a specific turn response does not contain forbidden text. +type TurnResponseNotContainsRule struct { + Turn int `json:"turn" yaml:"turn"` + NotContains []string `json:"not_contains" yaml:"not_contains"` +} + +// ToolCalledInTurnRule checks that a tool was called during a specific turn. +type ToolCalledInTurnRule struct { + Turn int `json:"turn" yaml:"turn"` + Name string `json:"name" yaml:"name"` + Args map[string]any `json:"args,omitempty" yaml:"args,omitempty"` +} + +// ToolNotCalledInTurnRule checks that a tool was NOT called during a specific turn. +type ToolNotCalledInTurnRule struct { + Turn int `json:"turn" yaml:"turn"` + Name string `json:"name" yaml:"name"` } // OutputContainsRule checks if output contains specific text. @@ -263,17 +294,28 @@ type Input struct { // Turn is a single conversation turn. type Turn struct { - Role string `yaml:"role"` // user - Content string `yaml:"content"` - PostCondition *PostCondition `yaml:"post_condition,omitempty"` + Role string `yaml:"role"` // user + Content string `yaml:"content"` + PostCondition *PostCondition `yaml:"post_condition,omitempty"` + Capture []CaptureRule `yaml:"capture,omitempty"` + TimeoutSeconds int `yaml:"timeout_seconds,omitempty"` } // PostCondition checks output after a turn. type PostCondition struct { MustContainAny []string `yaml:"must_contain_any,omitempty"` + MustContainAll []string `yaml:"must_contain_all,omitempty"` + MustNotContain []string `yaml:"must_not_contain,omitempty"` OnFail string `yaml:"on_fail,omitempty"` // skip_remaining, fail } +// CaptureRule defines how to extract a value from a turn response. +type CaptureRule struct { + Variable string `yaml:"variable"` + Pattern string `yaml:"pattern,omitempty"` + JSONPath string `yaml:"jsonpath,omitempty"` +} + // Context describes test case setup. type Context struct { RepoFixture string `yaml:"repo_fixture,omitempty"` diff --git a/internal/config/validator.go b/internal/config/validator.go index 2cec472f..0bfdcf61 100644 --- a/internal/config/validator.go +++ b/internal/config/validator.go @@ -3,6 +3,7 @@ package config import ( "fmt" "path" + "regexp" "slices" "strings" @@ -95,6 +96,7 @@ func (v *Validator) ValidateEvalConfig(cfg *EvalConfig) error { } errs = append(errs, validateCollectArtifacts("cases.defaults.collect_artifacts", cfg.Cases.Defaults.CollectArtifacts)...) + errs = append(errs, validateJudgeTypeAndLocalFields(cfg.Judge)...) if len(errs) > 0 { return fmt.Errorf("validation errors:\n - %s", strings.Join(errs, "\n - ")) @@ -110,25 +112,44 @@ func (v *Validator) ValidateCaseConfig(cfg *CaseConfig) error { // id is optional - Loader auto-generates from filename if not specified // See loader.go:LoadCaseConfig for the fallback logic - // input.prompt or input.turns is required + // input.prompt or input.turns is required, but not both if cfg.Input.Prompt == "" && len(cfg.Input.Turns) == 0 { errs = append(errs, "input.prompt or input.turns is required") } + if cfg.Input.Prompt != "" && len(cfg.Input.Turns) > 0 { + errs = append(errs, "input.prompt and input.turns are mutually exclusive") + } - // if turns is specified, each turn must have role and content + // if turns is specified, each turn must have role=user and content for i, turn := range cfg.Input.Turns { if turn.Role == "" { errs = append(errs, fmt.Sprintf("input.turns[%d].role is required", i)) + } else if turn.Role != "user" { + errs = append(errs, fmt.Sprintf("input.turns[%d].role must be \"user\", got %q", i, turn.Role)) } if turn.Content == "" { errs = append(errs, fmt.Sprintf("input.turns[%d].content is required", i)) } + if turn.TimeoutSeconds < 0 { + errs = append(errs, fmt.Sprintf("input.turns[%d].timeout_seconds must be non-negative", i)) + } + if turn.PostCondition != nil { + errs = append(errs, validatePostCondition(i, turn.PostCondition)...) + } + for j, cr := range turn.Capture { + errs = append(errs, validateCaptureRule(i, j, cr)...) + } } - // validate judge config if specified at case level - if cfg.Judge.Type != "" { - errs = append(errs, validateJudgeTypeAndFields(cfg.Judge)...) + // validate per-turn judge rules reference valid turn numbers + turnsTotal := len(cfg.Input.Turns) + if cfg.Judge.Type != "" || len(cfg.Judge.Skills) > 0 { + errs = append(errs, validateJudgeTypeAndLocalFields(cfg.Judge)...) } + var allRules []Rule + allRules = append(allRules, cfg.Judge.Success...) + allRules = append(allRules, cfg.Judge.Failure...) + errs = append(errs, validatePerTurnRules(allRules, turnsTotal)...) errs = append(errs, validateCollectArtifacts("collect_artifacts", cfg.CollectArtifacts)...) @@ -139,34 +160,51 @@ func (v *Validator) ValidateCaseConfig(cfg *CaseConfig) error { return nil } -// validateJudgeTypeAndFields validates judge type and its conditional fields. -func validateJudgeTypeAndFields(judge JudgeConfig) []string { +// validateJudgeTypeAndLocalFields validates judge fields that do not depend on inheritance. +func validateJudgeTypeAndLocalFields(judge JudgeConfig) []string { var errs []string if judge.Type != "" && !isValidJudgeType(judge.Type) { errs = append(errs, "judge.type must be one of: rule_based, script, agent_judge") } + if len(judge.Skills) > 0 && judge.Type != judgeTypeAgentJudge { + errs = append(errs, "judge.skills is only supported when judge.type is agent_judge") + } + errs = append(errs, validateSkillRefs("judge.skills", judge.Skills)...) + + errs = append(errs, validatePassThreshold(judge.PassThreshold)...) + + if judge.TimeoutSeconds != nil && *judge.TimeoutSeconds < 0 { + errs = append(errs, "judge.timeout_seconds must be non-negative") + } - // script type requires script_path + return errs +} + +func validateJudgeTypeAndRequiredFields(judge JudgeConfig) []string { + errs := validateJudgeTypeAndLocalFields(judge) if judge.Type == judgeTypeScript && judge.ScriptPath == "" { errs = append(errs, "judge.script_path is required when judge.type is script") } - - // agent_judge type requires model and criteria if judge.Type == judgeTypeAgentJudge && judge.Model == "" { errs = append(errs, "judge.model is required when judge.type is agent_judge") } - if judge.Type == judgeTypeAgentJudge && len(judge.Criteria) == 0 { errs = append(errs, "judge.criteria is required when judge.type is agent_judge") } + return errs +} - errs = append(errs, validatePassThreshold(judge.PassThreshold)...) - - if judge.TimeoutSeconds != nil && *judge.TimeoutSeconds < 0 { - errs = append(errs, "judge.timeout_seconds must be non-negative") +func validateSkillRefs(field string, refs []SkillRef) []string { + var errs []string + for i, ref := range refs { + if strings.TrimSpace(ref.Source) == "" { + errs = append(errs, fmt.Sprintf("%s[%d].source is required", field, i)) + } + if ref.Source == "local_path" && strings.TrimSpace(ref.Path) == "" { + errs = append(errs, fmt.Sprintf("%s[%d].path is required when source is local_path", field, i)) + } } - return errs } @@ -214,11 +252,32 @@ func (v *Validator) ValidateAll(result *EvalResult) error { if err := v.ValidateCaseConfig(c); err != nil { return fmt.Errorf("case %s: %w", c.ID, err) } + effectiveJudge := mergeJudgeConfigForValidation(result.Eval.Judge, c.Judge) + if errs := validateJudgeTypeAndRequiredFields(effectiveJudge); len(errs) > 0 { + return fmt.Errorf("case %s: validation errors:\n - %s", c.ID, strings.Join(errs, "\n - ")) + } } return nil } +func mergeJudgeConfigForValidation(global, caseLevel JudgeConfig) JudgeConfig { + if caseLevel.Type != "" { + merged := caseLevel + if merged.Model == "" { + merged.Model = global.Model + } + if merged.PassThreshold == nil { + merged.PassThreshold = global.PassThreshold + } + if merged.TimeoutSeconds == nil { + merged.TimeoutSeconds = global.TimeoutSeconds + } + return merged + } + return global +} + // validateEngine checks engine.custom against the Custom Engine contract. // A non-built-in engine.name requires an engine.custom block; a built-in // engine.name ignores engine.custom entirely. @@ -341,6 +400,130 @@ func isValidJudgeType(t string) bool { return t == judgeTypeRuleBased || t == judgeTypeScript || t == judgeTypeAgentJudge } +// captureVariablePattern is the allowed pattern for capture variable names. +var captureVariablePattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`) + +// validatePostCondition validates a turn's post_condition block. +func validatePostCondition(turnIdx int, pc *PostCondition) []string { + var errs []string + prefix := fmt.Sprintf("input.turns[%d].post_condition", turnIdx) + + // on_fail must be empty, "fail", or "skip_remaining" + if pc.OnFail != "" && pc.OnFail != "fail" && pc.OnFail != "skip_remaining" { + errs = append(errs, fmt.Sprintf("%s.on_fail must be empty, \"fail\", or \"skip_remaining\", got %q", prefix, pc.OnFail)) + } + + // Must include at least one check field + if len(pc.MustContainAny) == 0 && len(pc.MustContainAll) == 0 && len(pc.MustNotContain) == 0 { + errs = append(errs, prefix+" must include at least one of must_contain_any, must_contain_all, or must_not_contain") + } + + return errs +} + +// validateCaptureRule validates a single capture rule. +func validateCaptureRule(turnIdx, capIdx int, cr CaptureRule) []string { + var errs []string + prefix := fmt.Sprintf("input.turns[%d].capture[%d]", turnIdx, capIdx) + + // variable must match identifier pattern + if cr.Variable == "" { + errs = append(errs, prefix+".variable is required") + } else if !captureVariablePattern.MatchString(cr.Variable) { + errs = append(errs, fmt.Sprintf("%s.variable %q must match pattern [A-Za-z_][A-Za-z0-9_]*", prefix, cr.Variable)) + } + + // Exactly one extractor must be specified + hasPattern := cr.Pattern != "" + hasJSONPath := cr.JSONPath != "" + if !hasPattern && !hasJSONPath { + errs = append(errs, prefix+" must specify exactly one extractor: pattern or jsonpath") + } else if hasPattern && hasJSONPath { + errs = append(errs, prefix+" must specify exactly one extractor: pattern or jsonpath (both set)") + } + + // Regex pattern must compile + if hasPattern { + if _, err := regexp.Compile(cr.Pattern); err != nil { + errs = append(errs, fmt.Sprintf("%s.pattern is invalid regex: %v", prefix, err)) + } + } + + return errs +} + +// validatePerTurnRules validates per-turn judge rules reference valid turn numbers. +func validatePerTurnRules(rules []Rule, turnsTotal int) []string { + var errs []string + for _, rule := range rules { + if rule.TurnResponseContains != nil { + errs = append(errs, validateTurnResponseContainsRule(rule.TurnResponseContains, turnsTotal)...) + } + if rule.TurnResponseNotContains != nil { + errs = append(errs, validateTurnResponseNotContainsRule(rule.TurnResponseNotContains, turnsTotal)...) + } + if rule.ToolCalledInTurn != nil { + errs = append(errs, validateToolCalledInTurnRule(rule.ToolCalledInTurn, turnsTotal)...) + } + if rule.ToolNotCalledInTurn != nil { + errs = append(errs, validateToolNotCalledInTurnRule(rule.ToolNotCalledInTurn, turnsTotal)...) + } + } + return errs +} + +func validateTurnResponseContainsRule(r *TurnResponseContainsRule, turnsTotal int) []string { + var errs []string + if r.Turn < 1 { + errs = append(errs, "turn_response_contains.turn must be >= 1") + } else if turnsTotal > 0 && r.Turn > turnsTotal { + errs = append(errs, fmt.Sprintf("turn_response_contains.turn %d exceeds total turns %d", r.Turn, turnsTotal)) + } + if len(r.ContainsAll) == 0 && len(r.ContainsAny) == 0 { + errs = append(errs, "turn_response_contains must specify contains_all or contains_any") + } + return errs +} + +func validateTurnResponseNotContainsRule(r *TurnResponseNotContainsRule, turnsTotal int) []string { + var errs []string + if r.Turn < 1 { + errs = append(errs, "turn_response_not_contains.turn must be >= 1") + } else if turnsTotal > 0 && r.Turn > turnsTotal { + errs = append(errs, fmt.Sprintf("turn_response_not_contains.turn %d exceeds total turns %d", r.Turn, turnsTotal)) + } + if len(r.NotContains) == 0 { + errs = append(errs, "turn_response_not_contains.not_contains is required") + } + return errs +} + +func validateToolCalledInTurnRule(r *ToolCalledInTurnRule, turnsTotal int) []string { + var errs []string + if r.Turn < 1 { + errs = append(errs, "tool_called_in_turn.turn must be >= 1") + } else if turnsTotal > 0 && r.Turn > turnsTotal { + errs = append(errs, fmt.Sprintf("tool_called_in_turn.turn %d exceeds total turns %d", r.Turn, turnsTotal)) + } + if r.Name == "" { + errs = append(errs, "tool_called_in_turn.name is required") + } + return errs +} + +func validateToolNotCalledInTurnRule(r *ToolNotCalledInTurnRule, turnsTotal int) []string { + var errs []string + if r.Turn < 1 { + errs = append(errs, "tool_not_called_in_turn.turn must be >= 1") + } else if turnsTotal > 0 && r.Turn > turnsTotal { + errs = append(errs, fmt.Sprintf("tool_not_called_in_turn.turn %d exceeds total turns %d", r.Turn, turnsTotal)) + } + if r.Name == "" { + errs = append(errs, "tool_not_called_in_turn.name is required") + } + return errs +} + func validateNetworkPolicy(env Environment) []string { policy := env.NetworkPolicy if policy == "" { diff --git a/internal/config/validator_test.go b/internal/config/validator_test.go index ee7bf4f6..d37da8d8 100644 --- a/internal/config/validator_test.go +++ b/internal/config/validator_test.go @@ -335,7 +335,7 @@ func TestValidator_ValidateEvalConfig(t *testing.T) { errMsg: "cases.retry_policy.max_retries must be <=", }, { - name: "invalid judge.type at eval level is ignored", + name: "invalid judge.type at eval level is rejected", cfg: &EvalConfig{ SchemaVersion: "v1alpha1", Environment: Environment{Type: "none"}, @@ -353,10 +353,11 @@ func TestValidator_ValidateEvalConfig(t *testing.T) { Type: "invalid", }, }, - wantErr: false, + wantErr: true, + errMsg: "judge.type must be one of", }, { - name: "script without script_path at eval level is ignored", + name: "script without script_path at eval level is checked by ValidateAll", cfg: &EvalConfig{ SchemaVersion: "v1alpha1", Environment: Environment{Type: "none"}, @@ -377,7 +378,7 @@ func TestValidator_ValidateEvalConfig(t *testing.T) { wantErr: false, }, { - name: "agent_judge without model at eval level is ignored", + name: "agent_judge without model at eval level is checked by ValidateAll", cfg: &EvalConfig{ SchemaVersion: "v1alpha1", Environment: Environment{Type: "none"}, @@ -422,7 +423,7 @@ func TestValidator_ValidateEvalConfig(t *testing.T) { wantErr: false, }, { - name: "agent_judge with threshold above one at eval level is ignored", + name: "agent_judge with threshold above one at eval level is rejected", cfg: &EvalConfig{ SchemaVersion: "v1alpha1", Environment: Environment{Type: "none"}, @@ -443,7 +444,8 @@ func TestValidator_ValidateEvalConfig(t *testing.T) { PassThreshold: float64Ptr(1.1), }, }, - wantErr: false, + wantErr: true, + errMsg: "judge.pass_threshold must be between", }, { name: "valid network_policy deny_all with opensandbox", @@ -762,6 +764,18 @@ func TestValidator_ValidateCaseConfig(t *testing.T) { wantErr: true, errMsg: "input.prompt or input.turns is required", }, + { + name: "prompt and turns are mutually exclusive", + cfg: &CaseConfig{ + ID: "test-case", + Input: Input{ + Prompt: "Say hello", + Turns: []Turn{{Role: "user", Content: "Hi"}}, + }, + }, + wantErr: true, + errMsg: "input.prompt and input.turns are mutually exclusive", + }, { name: "turn with missing role", cfg: &CaseConfig{ @@ -775,6 +789,19 @@ func TestValidator_ValidateCaseConfig(t *testing.T) { wantErr: true, errMsg: "input.turns[0].role is required", }, + { + name: "turn with non-user role", + cfg: &CaseConfig{ + ID: "test-case", + Input: Input{ + Turns: []Turn{ + {Role: "assistant", Content: "Hello"}, + }, + }, + }, + wantErr: true, + errMsg: "input.turns[0].role must be \"user\"", + }, { name: "turn with missing content", cfg: &CaseConfig{ @@ -788,6 +815,19 @@ func TestValidator_ValidateCaseConfig(t *testing.T) { wantErr: true, errMsg: "input.turns[0].content is required", }, + { + name: "turn with negative timeout_seconds", + cfg: &CaseConfig{ + ID: "test-case", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hi", TimeoutSeconds: -1}, + }, + }, + }, + wantErr: true, + errMsg: "input.turns[0].timeout_seconds must be non-negative", + }, { name: "case-level agent_judge with negative threshold", cfg: &CaseConfig{ @@ -890,6 +930,131 @@ func TestValidator_ValidateAll(t *testing.T) { }) } +func TestValidator_JudgeSkills(t *testing.T) { + t.Parallel() + validator := NewValidator() + + base := func(judge JudgeConfig) *EvalConfig { + return &EvalConfig{ + SchemaVersion: "v1alpha1", + Environment: Environment{Type: "none"}, + Engine: EngineConfig{Name: "claude_code"}, + Cases: CasesConfig{Files: []string{"evals/cases/test.yaml"}}, + Judge: judge, + } + } + + tests := []struct { + name string + judge JudgeConfig + errMsg string + }{ + { + name: "agent_judge accepts skills", + judge: JudgeConfig{ + Type: "agent_judge", + Model: "test-model", + Criteria: []string{"criterion"}, + Skills: []SkillRef{{Source: "local_path", Path: "evals/fixtures/judge-skill"}}, + }, + }, + { + name: "rule_based rejects skills", + judge: JudgeConfig{ + Type: "rule_based", + Skills: []SkillRef{{Source: "local_path", Path: "evals/fixtures/judge-skill"}}, + }, + errMsg: "judge.skills is only supported", + }, + { + name: "local_path requires path", + judge: JudgeConfig{ + Type: "agent_judge", + Model: "test-model", + Criteria: []string{"criterion"}, + Skills: []SkillRef{{Source: "local_path"}}, + }, + errMsg: "judge.skills[0].path is required", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + err := validator.ValidateEvalConfig(base(tt.judge)) + if tt.errMsg == "" { + if err != nil { + t.Fatalf("ValidateEvalConfig() error = %v, want nil", err) + } + return + } + if err == nil || !strings.Contains(err.Error(), tt.errMsg) { + t.Fatalf("ValidateEvalConfig() error = %v, want containing %q", err, tt.errMsg) + } + }) + } +} + +func TestValidator_ValidateAll_CaseAgentJudgeInheritsGlobalModel(t *testing.T) { + t.Parallel() + validator := NewValidator() + + result := &EvalResult{ + Eval: &EvalConfig{ + SchemaVersion: "v1alpha1", + Environment: Environment{Type: "none"}, + Engine: EngineConfig{Name: "claude_code"}, + Cases: CasesConfig{Files: []string{"evals/cases/test.yaml"}}, + Judge: JudgeConfig{ + Type: "agent_judge", + Model: "global-model", + Criteria: []string{"global criterion"}, + }, + }, + Cases: []*CaseConfig{ + { + ID: "case-agent-judge", + Input: Input{Prompt: "test"}, + Judge: JudgeConfig{ + Type: "agent_judge", + Criteria: []string{"case criterion"}, + Skills: []SkillRef{{Source: "local_path", Path: "evals/fixtures/case-judge"}}, + }, + }, + }, + } + + if err := validator.ValidateAll(result); err != nil { + t.Fatalf("ValidateAll() error = %v, want nil", err) + } +} + +func TestValidator_ValidateAll_EffectiveAgentJudgeRequiresCriteria(t *testing.T) { + t.Parallel() + validator := NewValidator() + + result := &EvalResult{ + Eval: &EvalConfig{ + SchemaVersion: "v1alpha1", + Environment: Environment{Type: "none"}, + Engine: EngineConfig{Name: "claude_code"}, + Cases: CasesConfig{Files: []string{"evals/cases/test.yaml"}}, + Judge: JudgeConfig{ + Type: "agent_judge", + Model: "global-model", + }, + }, + Cases: []*CaseConfig{ + {ID: "case-agent-judge", Input: Input{Prompt: "test"}}, + }, + } + + err := validator.ValidateAll(result) + if err == nil || !strings.Contains(err.Error(), "judge.criteria is required") { + t.Fatalf("ValidateAll() error = %v, want missing criteria", err) + } +} + func contains(s, substr string) bool { return len(s) >= len(substr) && (s == substr || len(s) > 0 && containsHelper(s, substr)) } @@ -958,3 +1123,480 @@ func TestValidator_CollectArtifacts(t *testing.T) { } }) } + +//nolint:funlen // table-driven test +func TestValidator_PostCondition(t *testing.T) { + t.Parallel() + validator := NewValidator() + + tests := []struct { + name string + cfg *CaseConfig + wantErr bool + errMsg string + }{ + { + name: "valid post_condition with must_contain_any", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", PostCondition: &PostCondition{ + MustContainAny: []string{"hi", "hello"}, + }}, + }, + }, + }, + wantErr: false, + }, + { + name: "valid post_condition with must_contain_all", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", PostCondition: &PostCondition{ + MustContainAll: []string{"greeting", "response"}, + }}, + }, + }, + }, + wantErr: false, + }, + { + name: "valid post_condition with must_not_contain", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", PostCondition: &PostCondition{ + MustNotContain: []string{"error"}, + }}, + }, + }, + }, + wantErr: false, + }, + { + name: "valid post_condition with on_fail=fail", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", PostCondition: &PostCondition{ + MustContainAny: []string{"ok"}, + OnFail: "fail", + }}, + }, + }, + }, + wantErr: false, + }, + { + name: "valid post_condition with on_fail=skip_remaining", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", PostCondition: &PostCondition{ + MustContainAny: []string{"ok"}, + OnFail: "skip_remaining", + }}, + }, + }, + }, + wantErr: false, + }, + { + name: "post_condition with invalid on_fail", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", PostCondition: &PostCondition{ + MustContainAny: []string{"ok"}, + OnFail: "abort", + }}, + }, + }, + }, + wantErr: true, + errMsg: "on_fail must be empty", + }, + { + name: "post_condition with no check fields", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", PostCondition: &PostCondition{ + OnFail: "fail", + }}, + }, + }, + }, + wantErr: true, + errMsg: "must include at least one of must_contain_any, must_contain_all, or must_not_contain", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + err := validator.ValidateCaseConfig(tt.cfg) + if (err != nil) != tt.wantErr { + t.Errorf("error = %v, wantErr %v", err, tt.wantErr) + return + } + if tt.wantErr && err != nil && tt.errMsg != "" && !strings.Contains(err.Error(), tt.errMsg) { + t.Errorf("error = %v, should contain %q", err, tt.errMsg) + } + }) + } +} + +//nolint:funlen // table-driven test +func TestValidator_CaptureRules(t *testing.T) { + t.Parallel() + validator := NewValidator() + + tests := []struct { + name string + cfg *CaseConfig + wantErr bool + errMsg string + }{ + { + name: "valid capture with pattern", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", Capture: []CaptureRule{ + {Variable: "session_id", Pattern: `id:\s*(?P[a-f0-9]+)`}, + }}, + }, + }, + }, + wantErr: false, + }, + { + name: "valid capture with jsonpath", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", Capture: []CaptureRule{ + {Variable: "name", JSONPath: "$.result.name"}, + }}, + }, + }, + }, + wantErr: false, + }, + { + name: "capture with empty variable", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", Capture: []CaptureRule{ + {Variable: "", Pattern: `id:\s+(\w+)`}, + }}, + }, + }, + }, + wantErr: true, + errMsg: "variable is required", + }, + { + name: "capture with invalid variable name", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", Capture: []CaptureRule{ + {Variable: "123invalid", Pattern: `id:\s+(\w+)`}, + }}, + }, + }, + }, + wantErr: true, + errMsg: "must match pattern", + }, + { + name: "capture with no extractor", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", Capture: []CaptureRule{ + {Variable: "myvar"}, + }}, + }, + }, + }, + wantErr: true, + errMsg: "must specify exactly one extractor", + }, + { + name: "capture with both extractors", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", Capture: []CaptureRule{ + {Variable: "myvar", Pattern: `(\w+)`, JSONPath: "$.x"}, + }}, + }, + }, + }, + wantErr: true, + errMsg: "must specify exactly one extractor: pattern or jsonpath (both set)", + }, + { + name: "capture with invalid regex pattern", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello", Capture: []CaptureRule{ + {Variable: "myvar", Pattern: `[invalid`}, + }}, + }, + }, + }, + wantErr: true, + errMsg: "pattern is invalid regex", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + err := validator.ValidateCaseConfig(tt.cfg) + if (err != nil) != tt.wantErr { + t.Errorf("error = %v, wantErr %v", err, tt.wantErr) + return + } + if tt.wantErr && err != nil && tt.errMsg != "" && !strings.Contains(err.Error(), tt.errMsg) { + t.Errorf("error = %v, should contain %q", err, tt.errMsg) + } + }) + } +} + +//nolint:funlen // table-driven test +func TestValidator_PerTurnJudgeRules(t *testing.T) { + t.Parallel() + validator := NewValidator() + + tests := []struct { + name string + cfg *CaseConfig + wantErr bool + errMsg string + }{ + { + name: "valid turn_response_contains", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello"}, + {Role: "user", Content: "How are you?"}, + }, + }, + Judge: JudgeConfig{ + Type: "rule_based", + Success: []Rule{ + {TurnResponseContains: &TurnResponseContainsRule{Turn: 1, ContainsAll: []string{"hello"}}}, + }, + }, + }, + wantErr: false, + }, + { + name: "turn_response_contains with turn < 1", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello"}, + }, + }, + Judge: JudgeConfig{ + Type: "rule_based", + Success: []Rule{ + {TurnResponseContains: &TurnResponseContainsRule{Turn: 0, ContainsAll: []string{"x"}}}, + }, + }, + }, + wantErr: true, + errMsg: "turn_response_contains.turn must be >= 1", + }, + { + name: "turn_response_contains exceeds total turns", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello"}, + }, + }, + Judge: JudgeConfig{ + Type: "rule_based", + Success: []Rule{ + {TurnResponseContains: &TurnResponseContainsRule{Turn: 5, ContainsAll: []string{"x"}}}, + }, + }, + }, + wantErr: true, + errMsg: "exceeds total turns", + }, + { + name: "turn_response_contains without contains_all or contains_any", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello"}, + }, + }, + Judge: JudgeConfig{ + Type: "rule_based", + Success: []Rule{ + {TurnResponseContains: &TurnResponseContainsRule{Turn: 1}}, + }, + }, + }, + wantErr: true, + errMsg: "must specify contains_all or contains_any", + }, + { + name: "turn_response_not_contains without not_contains", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello"}, + }, + }, + Judge: JudgeConfig{ + Type: "rule_based", + Success: []Rule{ + {TurnResponseNotContains: &TurnResponseNotContainsRule{Turn: 1}}, + }, + }, + }, + wantErr: true, + errMsg: "turn_response_not_contains.not_contains is required", + }, + { + name: "tool_called_in_turn without name", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello"}, + }, + }, + Judge: JudgeConfig{ + Type: "rule_based", + Success: []Rule{ + {ToolCalledInTurn: &ToolCalledInTurnRule{Turn: 1}}, + }, + }, + }, + wantErr: true, + errMsg: "tool_called_in_turn.name is required", + }, + { + name: "tool_not_called_in_turn without name", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello"}, + }, + }, + Judge: JudgeConfig{ + Type: "rule_based", + Success: []Rule{ + {ToolNotCalledInTurn: &ToolNotCalledInTurnRule{Turn: 1}}, + }, + }, + }, + wantErr: true, + errMsg: "tool_not_called_in_turn.name is required", + }, + { + name: "tool_called_in_turn with turn < 1", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello"}, + }, + }, + Judge: JudgeConfig{ + Type: "rule_based", + Success: []Rule{ + {ToolCalledInTurn: &ToolCalledInTurnRule{Turn: 0, Name: "write_file"}}, + }, + }, + }, + wantErr: true, + errMsg: "tool_called_in_turn.turn must be >= 1", + }, + { + name: "valid per-turn rules in failure list", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Turns: []Turn{ + {Role: "user", Content: "Hello"}, + {Role: "user", Content: "Bye"}, + }, + }, + Judge: JudgeConfig{ + Type: "rule_based", + Failure: []Rule{ + {TurnResponseContains: &TurnResponseContainsRule{Turn: 2, ContainsAny: []string{"error"}}}, + }, + }, + }, + wantErr: false, + }, + { + name: "per-turn rules valid when no turns defined (prompt mode)", + cfg: &CaseConfig{ + ID: "test", + Input: Input{ + Prompt: "Hello", + }, + Judge: JudgeConfig{ + Type: "rule_based", + Success: []Rule{ + {TurnResponseContains: &TurnResponseContainsRule{Turn: 1, ContainsAll: []string{"x"}}}, + }, + }, + }, + wantErr: false, // no turns defined means turnsTotal=0, skip upper-bound check + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + err := validator.ValidateCaseConfig(tt.cfg) + if (err != nil) != tt.wantErr { + t.Errorf("error = %v, wantErr %v", err, tt.wantErr) + return + } + if tt.wantErr && err != nil && tt.errMsg != "" && !strings.Contains(err.Error(), tt.errMsg) { + t.Errorf("error = %v, should contain %q", err, tt.errMsg) + } + }) + } +} diff --git a/internal/evaluator/evaluator.go b/internal/evaluator/evaluator.go index bd790537..838af91e 100644 --- a/internal/evaluator/evaluator.go +++ b/internal/evaluator/evaluator.go @@ -35,6 +35,8 @@ var ( agentDetectWithInitParams = agent.DetectAgentWithInitParams ) +const judgeTypeAgentJudge = "agent_judge" + // ProgressObserver receives progress notifications during evaluation. // Implementations must be safe for concurrent use. type ProgressObserver interface { @@ -82,9 +84,16 @@ type EvalResult struct { // TurnsTotal is the total number of turns defined in the case. TurnsTotal int + // TurnResults holds per-turn outcomes for multi-turn evaluations. + // Nil for single-turn cases. + TurnResults []TurnResult + // Grading is the judge evaluation result (nil if judge was skipped). Grading *judge.Result + // JudgeSkills records judge Skills configured for agent_judge. + JudgeSkills []judge.SkillInfo + // ExpectResult is the expect pre-check result (nil if no expect block). ExpectResult *judge.ExpectResult @@ -381,6 +390,31 @@ func (e *defaultEvaluator) executeCaseOnce(ctx context.Context, caseCfg *config. judgeCfg := judge.MergeJudgeConfig(e.evalCfg.Judge, caseCfg.Judge) + // Multi-turn branch: delegate to dedicated engine when the case defines + // input.turns AND the agent supports session resumption. Agents that do + // not implement SessionResumer fall through to the existing single-shot + // path (all messages in one Run call), with an explicit warning. + if len(caseCfg.Input.Turns) > 0 { + if _, ok := runAgent.(agent.SessionResumer); ok { + agentExecOpts := agent.ExecOptions{ + ArtifactDir: e.prepareOutputDir(ctx, configName, caseCfg.ID, "agent/run"), + TimeoutSec: caseTimeoutSeconds(e.evalCfg, caseCfg), + AgentMetadata: &runtime.AgentMetadata{ + CaseID: caseCfg.ID, + Variant: configName, + MaxTurns: caseMaxTurns(e.evalCfg, caseCfg), + }, + } + return e.executeMultiTurnCase(ctx, rt, caseCfg, configName, runAgent, agentExecOpts, startTime, judgeCfg, &result) + } + logging.WarnContextf( + ctx, + "Evaluator: case %s defines input.turns but agent %s does not support session resumption; falling back to a single batch prompt", + caseCfg.ID, + runAgent.Name(), + ) + } + var cleanupArtifacts func() finalizeArtifacts := func(*agent.SessionResult) {} if judgeNeedsWorkspaceDiff(judgeCfg) { @@ -477,6 +511,7 @@ func (e *defaultEvaluator) evaluateCaseSession( WorkspaceDiff: sessionWorkspaceDiff(sessionResult), GeneratedFiles: sessionGeneratedFiles(sessionResult), SessionResult: sessionResult, + TurnResults: toJudgeTurnResults(result.TurnResults), } if failed := e.runExpectPreCheck(ctx, caseCfg, configName, judgeInput, turnsTotal, result); failed { @@ -570,6 +605,9 @@ func (e *defaultEvaluator) runJudgePhaseWithSpan( logging.DebugContextf(ctx, "Judge: case %s using %s", caseCfg.ID, judgeLabel(judgeCfg)) logging.DebugContextf(ctx, "Judge: case %s generated_files=%d workspace_diff=%t", caseCfg.ID, len(judgeInput.GeneratedFiles), judgeInput.WorkspaceDiff != "") + if judgeCfg.Type == judgeTypeAgentJudge { + result.JudgeSkills = judge.SkillInfosFromRefs(judgeCfg.Skills) + } j, err := e.newJudgeForCase(ctx, rt, judgeCfg, runAgent) if err != nil { @@ -585,7 +623,7 @@ func (e *defaultEvaluator) runJudgePhaseWithSpan( logging.DebugContextf(ctx, "Judge: case %s has no judge configured, default PASS", caseCfg.ID) return *result } - if judgeCfg.Type == "agent_judge" { + if judgeCfg.Type == judgeTypeAgentJudge { judgeInput.ArtifactDir = e.prepareOutputDir(ctx, configName, caseCfg.ID, "judge/run") } @@ -640,6 +678,9 @@ func (e *defaultEvaluator) newJudgeForCase( if err != nil { return nil, err } + if err := e.installJudgeSkills(ctx, rt, judgeCfg, judgeAgent); err != nil { + return nil, err + } j, err := judge.NewJudge(judgeCfg, judgeAgent, rt) if err != nil { @@ -649,8 +690,33 @@ func (e *defaultEvaluator) newJudgeForCase( return j, nil } +func (e *defaultEvaluator) installJudgeSkills(ctx context.Context, rt runtime.Runtime, judgeCfg config.JudgeConfig, judgeAgent agent.Agent) error { + if judgeCfg.Type != judgeTypeAgentJudge || len(judgeCfg.Skills) == 0 { + return nil + } + skillDir := e.resolvedSkillDir() + if strings.TrimSpace(skillDir) == "" { + return errors.New("judge.skills requires a loader or skill directory to resolve local paths") + } + for i, ref := range judgeCfg.Skills { + skillCfg := resolveSkillConfig(skillDir, ref) + if err := judgeAgent.InstallSkill(ctx, rt, skillCfg); err != nil { + return fmt.Errorf("failed to install judge skill judge.skills[%d].path=%q: %w", i, ref.Path, err) + } + logging.DebugContextf(ctx, "Evaluator: judge skill installed: %s", filepath.Base(skillCfg.Source)) + } + return nil +} + +func (e *defaultEvaluator) resolvedSkillDir() string { + if e.loader != nil { + return e.loader.SkillDir() + } + return e.skillDir +} + func (e *defaultEvaluator) resolveJudgeAgent(ctx context.Context, judgeCfg config.JudgeConfig, runAgent agent.Agent) (agent.Agent, error) { - if judgeCfg.Type != "agent_judge" { + if judgeCfg.Type != judgeTypeAgentJudge { return runAgent, nil } @@ -856,16 +922,11 @@ func (e *defaultEvaluator) setupCaseEnvironment(ctx context.Context, rt runtime. if configName != "without_skill" && e.loader != nil { for _, skillRef := range e.evalCfg.Skills { - skillSourceDir := e.loader.SkillDir() - skillSource := filepath.Join(skillSourceDir, skillRef.Path) - skillCfg := runtime.SkillConfig{ - Source: skillSource, - Target: skillRef.Target, - } + skillCfg := resolveSkillConfig(e.loader.SkillDir(), skillRef) if err := ag.InstallSkill(ctx, rt, skillCfg); err != nil { return fmt.Errorf("failed to install skill %s: %w", skillRef.Path, err) } - logging.DebugContextf(ctx, "Evaluator: skill installed: %s", filepath.Base(skillSource)) + logging.DebugContextf(ctx, "Evaluator: skill installed: %s", filepath.Base(skillCfg.Source)) } } @@ -879,6 +940,17 @@ func (e *defaultEvaluator) setupCaseEnvironment(ctx context.Context, rt runtime. return nil } +func resolveSkillConfig(skillDir string, ref config.SkillRef) runtime.SkillConfig { + source := ref.Path + if source != "" && !filepath.IsAbs(source) { + source = filepath.Join(skillDir, source) + } + return runtime.SkillConfig{ + Source: source, + Target: ref.Target, + } +} + func (e *defaultEvaluator) provisionMCPConfig() (runtime.MCPConfig, map[string]string, error) { e.mcpOnce.Do(func() { skillDir := e.skillDir @@ -933,7 +1005,7 @@ func normalizeSessionResult(sessionResult *agent.SessionResult) *agent.SessionRe } func judgeNeedsWorkspaceDiff(cfg config.JudgeConfig) bool { - return cfg.Type == "agent_judge" + return cfg.Type == judgeTypeAgentJudge } func judgeLabel(cfg config.JudgeConfig) string { @@ -992,6 +1064,25 @@ func sessionGeneratedFiles(sessionResult *agent.SessionResult) []string { return sessionResult.Artifacts.GeneratedFiles } +// toJudgeTurnResults converts evaluator TurnResults to judge InputTurnResults. +func toJudgeTurnResults(turns []TurnResult) []judge.InputTurnResult { + if len(turns) == 0 { + return nil + } + out := make([]judge.InputTurnResult, len(turns)) + for i, tr := range turns { + out[i] = judge.InputTurnResult{ + TurnNumber: tr.TurnNumber, + Content: tr.Content, + Response: tr.Response, + Transcript: tr.Transcript, + Status: string(tr.Status), + Reason: tr.Reason, + } + } + return out +} + func prepareWorkspaceDiffState(ctx context.Context, rt runtime.Runtime, gitCtx *config.GitContext) (workspaceDiffState, error) { if gitCtx == nil || !gitCtx.Init { const probeScript = `if git rev-parse --git-dir >/dev/null 2>&1; then diff --git a/internal/evaluator/evaluator_test.go b/internal/evaluator/evaluator_test.go index 6dfe377c..6342350a 100644 --- a/internal/evaluator/evaluator_test.go +++ b/internal/evaluator/evaluator_test.go @@ -37,6 +37,7 @@ type mockAgent struct { output string err error credErr error + skillErr error runFunc func(ctx context.Context, rt runtime.Runtime, opts agent.ExecOptions, messages []transcript.Message) (*agent.SessionResult, error) runCall atomic.Int32 credCall atomic.Int32 @@ -46,6 +47,7 @@ type mockAgent struct { mu sync.Mutex lastMessages []transcript.Message lastSkill runtime.SkillConfig + skills []runtime.SkillConfig } func (m *mockAgent) Name() string { return m.name } @@ -63,8 +65,9 @@ func (m *mockAgent) InstallSkill(_ context.Context, _ runtime.Runtime, cfg runti m.skillCall.Add(1) m.mu.Lock() m.lastSkill = cfg + m.skills = append(m.skills, cfg) m.mu.Unlock() - return nil + return m.skillErr } func (m *mockAgent) Check(_ context.Context, _ runtime.Runtime) error { return nil } func (m *mockAgent) CheckCredentials(_ context.Context) error { @@ -276,6 +279,34 @@ func TestEvaluatorInputHelpers(t *testing.T) { } } +func TestExecuteCaseWarnsWhenTurnsFallbackToSingleBatch(t *testing.T) { + ag := &mockAgent{name: "batch-only", output: "batched response"} + rt := &mockRuntime{workspace: t.TempDir()} + e := newTestEvaluator(EvalOptions{ + Agent: ag, + EvalCfg: &config.EvalConfig{}, + }) + caseCfg := &config.CaseConfig{ + ID: "turns-fallback", + Input: config.Input{ + Turns: []config.Turn{ + {Role: "user", Content: "first"}, + {Role: "user", Content: "second"}, + }, + }, + } + + output := captureStdout(t, func() { + _ = e.executeCase(context.Background(), caseCfg, "with_skill", rt, nil) + }) + if !strings.Contains(output, "does not support session resumption") { + t.Fatalf("expected fallback warning, got %q", output) + } + if ag.runCall.Load() != 1 { + t.Fatalf("fallback should run the agent once, got %d calls", ag.runCall.Load()) + } +} + func TestEvaluatorRetryAndRecoveryHelpers(t *testing.T) { t.Parallel() @@ -1003,6 +1034,119 @@ func TestExecuteCase_AgentTimeoutDoesNotInvokeAgentJudge(t *testing.T) { } } +func TestExecuteCase_InstallsJudgeSkillsOnJudgeAgentOnly(t *testing.T) { + skillDir := t.TempDir() + judgeAgent := &mockAgent{ + name: "judge", + output: `{"results":[{"criterion":"uses rubric","passed":true,"evidence":"rubric applied"}]}`, + } + runAgent := &mockAgent{name: "run", output: "main response"} + + origDetect := agentDetectWithInitParams + agentDetectWithInitParams = func(_ string, _ credential.AgentInitParams, _ map[string]string) (agent.Agent, error) { + return judgeAgent, nil + } + defer func() { agentDetectWithInitParams = origDetect }() + + e := newTestEvaluator(EvalOptions{ + SkillDir: skillDir, + Agent: runAgent, + EvalCfg: &config.EvalConfig{ + Engine: config.EngineConfig{Name: "mock"}, + Judge: config.JudgeConfig{ + Type: "agent_judge", + Model: "judge-model", + Criteria: []string{"uses rubric"}, + Skills: []config.SkillRef{ + {Source: "local_path", Path: "evals/fixtures/judge-skill", Target: "~/.claude/skills/judge-skill"}, + {Source: "local_path", Path: "evals/fixtures/security-judge"}, + }, + }, + }, + }) + + result := e.executeCase( + context.Background(), + &config.CaseConfig{ID: "case-judge-skills", Input: config.Input{Prompt: "hello"}}, + "without_skill", + &mockRuntime{workspace: t.TempDir()}, + runAgent, + ) + + if result.Status != judge.StatusPass { + t.Fatalf("status = %s, err=%v", result.Status, result.Error) + } + if got := runAgent.skillCall.Load(); got != 0 { + t.Fatalf("run agent InstallSkill calls = %d, want 0", got) + } + if got := judgeAgent.skillCall.Load(); got != 2 { + t.Fatalf("judge agent InstallSkill calls = %d, want 2", got) + } + if len(judgeAgent.skills) != 2 { + t.Fatalf("judge installed skills = %#v", judgeAgent.skills) + } + wantFirst := filepath.Join(skillDir, "evals/fixtures/judge-skill") + if judgeAgent.skills[0].Source != wantFirst { + t.Fatalf("first judge skill source = %q, want %q", judgeAgent.skills[0].Source, wantFirst) + } + if judgeAgent.skills[0].Target != "~/.claude/skills/judge-skill" { + t.Fatalf("first judge skill target = %q", judgeAgent.skills[0].Target) + } + if len(result.JudgeSkills) != 2 || result.JudgeSkills[0].Path != "evals/fixtures/judge-skill" { + t.Fatalf("result JudgeSkills = %#v", result.JudgeSkills) + } + if len(judgeAgent.lastMessages) != 1 || !strings.Contains(judgeAgent.lastMessages[0].Content, "Mandatory Judge Skill Use") { + t.Fatalf("judge prompt missing mandatory skill use: %#v", judgeAgent.lastMessages) + } +} + +func TestExecuteCase_JudgeSkillInstallFailureReturnsError(t *testing.T) { + judgeAgent := &mockAgent{ + name: "judge", + skillErr: errors.New("install unsupported"), + output: `{"results":[{"criterion":"uses rubric","passed":true,"evidence":"rubric applied"}]}`, + } + runAgent := &mockAgent{name: "run", output: "main response"} + + origDetect := agentDetectWithInitParams + agentDetectWithInitParams = func(_ string, _ credential.AgentInitParams, _ map[string]string) (agent.Agent, error) { + return judgeAgent, nil + } + defer func() { agentDetectWithInitParams = origDetect }() + + e := newTestEvaluator(EvalOptions{ + SkillDir: t.TempDir(), + Agent: runAgent, + EvalCfg: &config.EvalConfig{ + Engine: config.EngineConfig{Name: "mock"}, + Judge: config.JudgeConfig{ + Type: "agent_judge", + Model: "judge-model", + Criteria: []string{"uses rubric"}, + Skills: []config.SkillRef{{Source: "local_path", Path: "evals/fixtures/judge-skill"}}, + }, + }, + }) + + result := e.executeCase( + context.Background(), + &config.CaseConfig{ID: "case-judge-skill-error", Input: config.Input{Prompt: "hello"}}, + "with_skill", + &mockRuntime{workspace: t.TempDir()}, + runAgent, + ) + + if result.Status != judge.StatusError { + t.Fatalf("status = %s, want ERROR", result.Status) + } + if result.Error == nil || !strings.Contains(result.Error.Error(), `judge.skills[0].path="evals/fixtures/judge-skill"`) { + t.Fatalf("error = %v, want judge skill path context", result.Error) + } + if got := judgeAgent.runCall.Load(); got != 0 { + t.Fatalf("judge Run calls = %d, want 0 after install failure", got) + } +} + func TestExecuteCase_NoJudge_DefaultPass(t *testing.T) { e := newTestEvaluator(EvalOptions{ Agent: &mockAgent{name: "test", output: "hello world"}, diff --git a/internal/evaluator/multiturn.go b/internal/evaluator/multiturn.go new file mode 100644 index 00000000..bc8a1339 --- /dev/null +++ b/internal/evaluator/multiturn.go @@ -0,0 +1,712 @@ +package evaluator + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "maps" + "regexp" + "strconv" + "strings" + "time" + + "github.com/alibaba/skill-up/internal/agent" + "github.com/alibaba/skill-up/internal/config" + "github.com/alibaba/skill-up/internal/judge" + "github.com/alibaba/skill-up/internal/logging" + "github.com/alibaba/skill-up/internal/runtime" + "github.com/alibaba/skill-up/pkg/transcript" +) + +// TurnStatus describes the outcome of a single conversation turn. +type TurnStatus string + +const ( + // TurnCompleted indicates the turn executed successfully. + TurnCompleted TurnStatus = "completed" + // TurnSkipped indicates the turn was skipped due to a prior post_condition failure. + TurnSkipped TurnStatus = "skipped" + // TurnFailed indicates the turn's post_condition failed with on_fail=fail. + TurnFailed TurnStatus = "failed" + // TurnError indicates a runtime or agent execution error. + TurnError TurnStatus = "error" +) + +// TurnResult holds the outcome of a single conversation turn during multi-turn evaluation. +type TurnResult struct { + // TurnNumber is the 1-based index of this turn. + TurnNumber int + // Content is the user message sent to the agent (after template substitution). + Content string + // Response is the assistant response text for this turn. + Response string + // Transcript is the per-turn interaction record returned by the agent. + Transcript transcript.Transcript + // SessionResult is the raw agent result for this turn. + SessionResult *agent.SessionResult + // Status is the outcome of this turn. + Status TurnStatus + // Reason describes why the turn was skipped/failed/errored. + Reason string + // CapturedVars holds variables captured from the turn response. + CapturedVars map[string]string +} + +// multiTurnState holds mutable state for a multi-turn execution. +type multiTurnState struct { + capturedVars map[string]string + sessionID string + turnResults []TurnResult + transcript transcript.Transcript + inputTokens int + outputTokens int + tokenBaseIn int + tokenBaseOut int + durationMs int64 +} + +// executeMultiTurn orchestrates multi-turn evaluation for cases with +// len(input.turns) > 1. It calls the agent once per turn, evaluating +// post_conditions and capturing variables between turns. +func (e *defaultEvaluator) executeMultiTurn( + ctx context.Context, + rt runtime.Runtime, + caseCfg *config.CaseConfig, + runAgent agent.Agent, + agentExecOpts agent.ExecOptions, +) ([]TurnResult, *agent.SessionResult, error) { + resumer, hasResumer := runAgent.(agent.SessionResumer) + if !hasResumer { + return nil, nil, fmt.Errorf("agent %s does not implement SessionResumer for multi-turn evaluation", runAgent.Name()) + } + + turns := caseCfg.Input.Turns + state := &multiTurnState{ + capturedVars: make(map[string]string), + turnResults: make([]TurnResult, 0, len(turns)), + } + + var lastSessionResult *agent.SessionResult + + for i, turn := range turns { + turnNum := i + 1 + + // Template substitution. + content, err := substituteTemplate(turn.Content, state.capturedVars) + if err != nil { + tr := TurnResult{TurnNumber: turnNum, Content: turn.Content, Status: TurnError, Reason: err.Error()} + state.turnResults = append(state.turnResults, tr) + return state.turnResults, lastSessionResult, fmt.Errorf("turn %d template substitution: %w", turnNum, err) + } + + // Execute the turn via the resumer. + sessionResult, execErr := executeSingleTurn(ctx, resumer, rt, agentExecOpts, content, turnNum, turn.TimeoutSeconds, state.sessionID) + if execErr != nil { + tr := TurnResult{TurnNumber: turnNum, Content: content, Status: TurnError, Reason: execErr.Error(), SessionResult: sessionResult} + state.turnResults = append(state.turnResults, tr) + return state.turnResults, buildAggregateResult(state, lastSessionResult), execErr + } + + // Update state from the successful turn. + if sessionResult != nil && sessionResult.SessionID != "" { + state.sessionID = sessionResult.SessionID + } + lastSessionResult = sessionResult + sessionResult, cumulativeUsage := normalizeTurnSessionResult(sessionResult, turnNum) + if cumulativeUsage { + accumulateCumulativeMetrics(state, sessionResult) + } else { + accumulatePerTurnMetrics(state, sessionResult) + } + + // Build turn result. + tr := buildTurnResult(turnNum, content, sessionResult) + if tr.Transcript != nil { + state.transcript = append(state.transcript, tr.Transcript...) + } + + // Evaluate post_condition. + if failed, earlyReturn := e.handlePostCondition(ctx, caseCfg.ID, turnNum, i, &tr, turn, turns, state); earlyReturn { + return state.turnResults, buildAggregateResult(state, lastSessionResult), nil + } else if failed { + return state.turnResults, buildAggregateResult(state, lastSessionResult), nil + } + + // Capture variables. + if len(turn.Capture) > 0 { + captured, captureErr := captureVariables(turn.Capture, tr.Response) + if captureErr != nil { + tr.Status = TurnError + tr.Reason = captureErr.Error() + state.turnResults = append(state.turnResults, tr) + return state.turnResults, buildAggregateResult(state, lastSessionResult), captureErr + } + tr.CapturedVars = captured + maps.Copy(state.capturedVars, captured) + } + + state.turnResults = append(state.turnResults, tr) + } + + return state.turnResults, buildAggregateResult(state, lastSessionResult), nil +} + +// executeSingleTurn invokes the resumer for one turn, applying an optional per-turn timeout. +func executeSingleTurn( + ctx context.Context, + resumer agent.SessionResumer, + rt runtime.Runtime, + opts agent.ExecOptions, + content string, + turnNum int, + timeoutSec int, + sessionID string, +) (*agent.SessionResult, error) { + msg := transcript.Message{Role: transcript.RoleUser, Content: content, Turn: turnNum} + + turnCtx := ctx + var cancel context.CancelFunc + if timeoutSec > 0 { + turnCtx, cancel = context.WithTimeout(ctx, time.Duration(timeoutSec)*time.Second) + } else { + cancel = func() {} + } + defer cancel() + + return resumer.RunTurn(turnCtx, rt, opts, msg, sessionID) +} + +// buildTurnResult creates a TurnResult from a successful agent execution. +func buildTurnResult(turnNum int, content string, sessionResult *agent.SessionResult) TurnResult { + tr := TurnResult{ + TurnNumber: turnNum, + Content: content, + SessionResult: sessionResult, + Status: TurnCompleted, + } + if sessionResult != nil { + tr.Response = sessionResult.FinalMessage + tr.Transcript = sessionResult.Transcript + } + return tr +} + +func normalizeTurnSessionResult(sessionResult *agent.SessionResult, turnNum int) (*agent.SessionResult, bool) { + if sessionResult == nil { + return nil, false + } + normalized := *sessionResult + turnTranscript, cumulative := transcriptForLogicalTurn(sessionResult.Transcript, turnNum) + normalized.Transcript = turnTranscript + if len(turnTranscript) > 0 { + normalized.Turns = 1 + if final := turnTranscript.FinalAssistantMessage(); final != "" { + normalized.FinalMessage = final + } + } + return &normalized, cumulative || sessionResult.Turns > 1 +} + +func transcriptForLogicalTurn(trans transcript.Transcript, turnNum int) (transcript.Transcript, bool) { + if len(trans) == 0 { + return nil, false + } + + matching := trans.MessagesForTurn(turnNum) + if len(matching) > 0 { + return matching, len(matching) < len(trans) + } + + normalized := make(transcript.Transcript, len(trans)) + copy(normalized, trans) + for i := range normalized { + normalized[i].Turn = turnNum + } + return normalized, false +} + +// substituteTemplate replaces {{variable}} placeholders in content with captured values. +// Unresolved placeholders are treated as errors. +func substituteTemplate(content string, vars map[string]string) (string, error) { + re := regexp.MustCompile(`\{\{(\w+)\}\}`) + var unresolved []string + result := re.ReplaceAllStringFunc(content, func(match string) string { + name := match[2 : len(match)-2] + if val, ok := vars[name]; ok { + return val + } + unresolved = append(unresolved, name) + return match + }) + if len(unresolved) > 0 { + return "", fmt.Errorf("unresolved template variable(s): %s", strings.Join(unresolved, ", ")) + } + return result, nil +} + +// evaluatePostCondition checks the response against the post_condition rules. +// Returns empty string on success; otherwise a human-readable failure reason. +func evaluatePostCondition(pc *config.PostCondition, response string) string { + if pc == nil { + return "" + } + + // must_contain_all: all required strings must appear. + for _, required := range pc.MustContainAll { + if !strings.Contains(response, required) { + return fmt.Sprintf("must_contain_all: missing %q", required) + } + } + + // must_contain_any: at least one string must appear. + if len(pc.MustContainAny) > 0 { + found := false + for _, candidate := range pc.MustContainAny { + if strings.Contains(response, candidate) { + found = true + break + } + } + if !found { + return fmt.Sprintf("must_contain_any: none of %v found", pc.MustContainAny) + } + } + + // must_not_contain: none may appear. + for _, forbidden := range pc.MustNotContain { + if strings.Contains(response, forbidden) { + return fmt.Sprintf("must_not_contain: found %q", forbidden) + } + } + + return "" +} + +// captureVariables extracts values from the response using the configured capture rules. +func captureVariables(rules []config.CaptureRule, response string) (map[string]string, error) { + captured := make(map[string]string, len(rules)) + for _, rule := range rules { + value, extractErr := captureVariableValue(rule, response) + if extractErr != nil { + return nil, fmt.Errorf("capture variable %q: %w", rule.Variable, extractErr) + } + if value == "" { + return nil, fmt.Errorf("capture variable %q: extractor matched but produced empty value", rule.Variable) + } + captured[rule.Variable] = value + } + return captured, nil +} + +func captureVariableValue(rule config.CaptureRule, response string) (string, error) { + switch { + case rule.Pattern != "": + re, err := regexp.Compile(rule.Pattern) + if err != nil { + return "", fmt.Errorf("invalid regex %q: %w", rule.Pattern, err) + } + match := re.FindStringSubmatch(response) + if match == nil { + return "", fmt.Errorf("pattern %q did not match response", rule.Pattern) + } + return extractCaptureValue(re, match) + case rule.JSONPath != "": + return extractJSONPathValue(response, rule.JSONPath) + default: + return "", errors.New("empty extractor") + } +} + +type jsonPathSegment struct { + key string + index *int +} + +func extractJSONPathValue(response, path string) (string, error) { + var doc any + decoder := json.NewDecoder(strings.NewReader(response)) + decoder.UseNumber() + if err := decoder.Decode(&doc); err != nil { + return "", fmt.Errorf("invalid JSON response for jsonpath %q: %w", path, err) + } + + segments, err := parseSimpleJSONPath(path) + if err != nil { + return "", err + } + current := doc + for _, segment := range segments { + if segment.index != nil { + arr, ok := current.([]any) + if !ok { + return "", fmt.Errorf("jsonpath %q expected array before [%d]", path, *segment.index) + } + if *segment.index < 0 || *segment.index >= len(arr) { + return "", fmt.Errorf("jsonpath %q index [%d] out of range", path, *segment.index) + } + current = arr[*segment.index] + continue + } + obj, ok := current.(map[string]any) + if !ok { + return "", fmt.Errorf("jsonpath %q expected object before %q", path, segment.key) + } + next, ok := obj[segment.key] + if !ok { + return "", fmt.Errorf("jsonpath %q key %q not found", path, segment.key) + } + current = next + } + + return jsonPathValueString(current) +} + +func parseSimpleJSONPath(path string) ([]jsonPathSegment, error) { + if path == "$" { + return nil, nil + } + if !strings.HasPrefix(path, "$") { + return nil, fmt.Errorf("jsonpath %q must start with $", path) + } + var segments []jsonPathSegment + for i := 1; i < len(path); { + switch path[i] { + case '.': + start := i + 1 + i = start + for i < len(path) && isJSONPathKeyChar(path[i]) { + i++ + } + if i == start { + return nil, fmt.Errorf("jsonpath %q has empty object key", path) + } + segments = append(segments, jsonPathSegment{key: path[start:i]}) + case '[': + end := strings.IndexByte(path[i:], ']') + if end < 0 { + return nil, fmt.Errorf("jsonpath %q has unterminated bracket", path) + } + end += i + raw := path[i+1 : end] + if raw == "" { + return nil, fmt.Errorf("jsonpath %q has empty bracket", path) + } + if quoted, ok := unquoteJSONPathKey(raw); ok { + segments = append(segments, jsonPathSegment{key: quoted}) + } else { + idx, err := strconv.Atoi(raw) + if err != nil { + return nil, fmt.Errorf("jsonpath %q supports only numeric indexes or quoted keys in brackets", path) + } + segments = append(segments, jsonPathSegment{index: &idx}) + } + i = end + 1 + default: + return nil, fmt.Errorf("jsonpath %q has unsupported token at %q", path, path[i:]) + } + } + return segments, nil +} + +func isJSONPathKeyChar(ch byte) bool { + return ch == '_' || ch == '-' || ch >= 'a' && ch <= 'z' || ch >= 'A' && ch <= 'Z' || ch >= '0' && ch <= '9' +} + +func unquoteJSONPathKey(raw string) (string, bool) { + if len(raw) < 2 { + return "", false + } + quote := raw[0] + if quote != '\'' && quote != '"' || raw[len(raw)-1] != quote { + return "", false + } + inner := raw[1 : len(raw)-1] + if strings.ContainsAny(inner, `'"[]`) { + return "", false + } + return inner, true +} + +func jsonPathValueString(value any) (string, error) { + switch v := value.(type) { + case nil: + return "", errors.New("jsonpath resolved to null") + case string: + return v, nil + case json.Number: + return v.String(), nil + case bool: + return strconv.FormatBool(v), nil + default: + data, err := json.Marshal(v) + if err != nil { + return "", err + } + return string(data), nil + } +} + +// extractCaptureValue retrieves the captured value from a regex match. +// It prefers a named group "value"; if absent, requires exactly one capture +// group (ambiguity from multiple unnamed groups is an error). +func extractCaptureValue(re *regexp.Regexp, match []string) (string, error) { + // Try named group "value" first. + for i, name := range re.SubexpNames() { + if name == "value" && i < len(match) { + return match[i], nil + } + } + // Count unnamed capture groups (SubexpNames()[0] is always the full match, skip it). + unnamedCount := countUnnamedGroups(re) + if unnamedCount == 0 { + return "", errors.New("pattern has no capture groups; use (?P...) or a single (...)") + } + if unnamedCount > 1 { + return "", fmt.Errorf("pattern has %d unnamed capture groups; use (?P...) to disambiguate", unnamedCount) + } + // Exactly one unnamed group — use first non-empty submatch. + if len(match) > 1 { + return match[1], nil + } + return "", nil +} + +// countUnnamedGroups counts capture groups that do not have a name. +func countUnnamedGroups(re *regexp.Regexp) int { + count := 0 + for i, name := range re.SubexpNames() { + if i == 0 { + continue // skip full match + } + if name == "" { + count++ + } + } + return count +} + +// handlePostCondition evaluates a turn's post_condition and handles failure. +// Returns (failed, earlyReturn). earlyReturn=true means skip_remaining was triggered. +func (e *defaultEvaluator) handlePostCondition( + ctx context.Context, + caseID string, + turnNum, turnIdx int, + tr *TurnResult, + turn config.Turn, + turns []config.Turn, + state *multiTurnState, +) (failed bool, earlyReturn bool) { + if turn.PostCondition == nil { + return false, false + } + reason := evaluatePostCondition(turn.PostCondition, tr.Response) + if reason == "" { + return false, false + } + + onFail := turn.PostCondition.OnFail + if onFail == "" { + onFail = "fail" + } + tr.Status = TurnFailed + tr.Reason = reason + state.turnResults = append(state.turnResults, *tr) + logging.DebugContextf(ctx, "Evaluator: case %s turn %d post_condition failed: %s (on_fail=%s)", caseID, turnNum, reason, onFail) + + if onFail == "skip_remaining" { + for j := turnIdx + 1; j < len(turns); j++ { + state.turnResults = append(state.turnResults, TurnResult{ + TurnNumber: j + 1, + Content: turns[j].Content, + Status: TurnSkipped, + Reason: fmt.Sprintf("skipped: turn %d post_condition failed", turnNum), + }) + } + return true, true + } + // on_fail == "fail" + return true, false +} + +// accumulatePerTurnMetrics adds metrics already scoped to the current turn. +func accumulatePerTurnMetrics(state *multiTurnState, result *agent.SessionResult) { + if result == nil { + return + } + state.inputTokens += result.InputTokens + state.outputTokens += result.OutputTokens + state.tokenBaseIn += result.InputTokens + state.tokenBaseOut += result.OutputTokens + state.durationMs += result.DurationMs +} + +// accumulateCumulativeMetrics converts cumulative session usage into per-turn deltas. +func accumulateCumulativeMetrics(state *multiTurnState, result *agent.SessionResult) { + if result == nil { + return + } + state.inputTokens += max(result.InputTokens-state.tokenBaseIn, 0) + state.outputTokens += max(result.OutputTokens-state.tokenBaseOut, 0) + state.tokenBaseIn = max(state.tokenBaseIn, result.InputTokens) + state.tokenBaseOut = max(state.tokenBaseOut, result.OutputTokens) + state.durationMs += result.DurationMs +} + +// buildAggregateResult constructs a SessionResult that aggregates all turns. +func buildAggregateResult(state *multiTurnState, lastResult *agent.SessionResult) *agent.SessionResult { + if lastResult == nil { + return &agent.SessionResult{ + Transcript: state.transcript, + InputTokens: state.inputTokens, + OutputTokens: state.outputTokens, + DurationMs: state.durationMs, + Turns: countCompletedTurns(state.turnResults), + } + } + agg := *lastResult + agg.Transcript = state.transcript + agg.InputTokens = state.inputTokens + agg.OutputTokens = state.outputTokens + agg.DurationMs = state.durationMs + agg.Turns = countCompletedTurns(state.turnResults) + // FinalMessage is the response from the last completed turn. + if msg := lastCompletedResponse(state.turnResults); msg != "" { + agg.FinalMessage = msg + } + return &agg +} + +// countCompletedTurns returns the number of turns that reached an agent response. +func countCompletedTurns(results []TurnResult) int { + n := 0 + for _, r := range results { + if r.Status == TurnCompleted || r.Status == TurnFailed { + n++ + } + } + return n +} + +// lastCompletedResponse returns the Response from the last TurnCompleted result. +func lastCompletedResponse(results []TurnResult) string { + for i := len(results) - 1; i >= 0; i-- { + if results[i].Status == TurnCompleted { + return results[i].Response + } + } + return "" +} + +// multiTurnStatus determines the overall case status from turn results. +func multiTurnStatus(results []TurnResult) judge.Status { + for i, r := range results { + switch r.Status { + case TurnError: + return judge.StatusError + case TurnFailed: + // Distinguish hard failure (on_fail=fail) from skip_remaining. + // on_fail=skip_remaining fills subsequent turns as TurnSkipped; + // if skipped entries follow the failure, proceed to judge. + hasSkippedAfter := false + for j := i + 1; j < len(results); j++ { + if results[j].Status == TurnSkipped { + hasSkippedAfter = true + break + } + } + if !hasSkippedAfter { + return judge.StatusFail + } + // skip_remaining: fall through to judge evaluation. + return "" + } + } + // If all remaining are completed/skipped, check if any completed. + hasCompleted := false + allSkipped := true + for _, r := range results { + if r.Status == TurnCompleted { + hasCompleted = true + allSkipped = false + } + if r.Status != TurnSkipped { + allSkipped = false + } + } + if allSkipped { + return judge.StatusSkip + } + if hasCompleted { + return "" // proceed to judge evaluation + } + return judge.StatusError +} + +// executeMultiTurnCase is the top-level entry point that executeCaseOnce +// delegates to when a case has multiple turns. It handles the full lifecycle: +// agent execution, post-condition gates, result aggregation, and judge phase. +func (e *defaultEvaluator) executeMultiTurnCase( + ctx context.Context, + rt runtime.Runtime, + caseCfg *config.CaseConfig, + configName string, + runAgent agent.Agent, + agentExecOpts agent.ExecOptions, + startTime time.Time, + judgeCfg config.JudgeConfig, + result *EvalResult, +) EvalResult { + // Workspace diff hooks. + var cleanupArtifacts func() + finalizeArtifacts := func(*agent.SessionResult) {} + if judgeNeedsWorkspaceDiff(judgeCfg) { + cleanupArtifacts, finalizeArtifacts = e.prepareWorkspaceArtifacts(ctx, rt, caseCfg) + defer cleanupArtifacts() + } + + turnResults, aggregateSession, execErr := e.executeMultiTurn(ctx, rt, caseCfg, runAgent, agentExecOpts) + result.TurnResults = turnResults + + finalizeArtifacts(aggregateSession) + result.SessionResult = normalizeSessionResult(aggregateSession) + + // Collect glob artifacts unconditionally. + e.collectGlobArtifacts(ctx, rt, configName, caseCfg) + + if execErr != nil { + if result.DurationMs == 0 { + result.DurationMs = time.Since(startTime).Milliseconds() + } + result.Status = judge.StatusError + result.Error = fmt.Errorf("agent execution failed: %w", execErr) + result.Configuration = configName + return *result + } + + // Determine status from turn results. + status := multiTurnStatus(turnResults) + switch status { + case judge.StatusError: + result.Status = judge.StatusError + result.Configuration = configName + return *result + case judge.StatusFail: + // Post-condition failure — mark FAIL, skip judge. + if result.DurationMs == 0 { + result.DurationMs = time.Since(startTime).Milliseconds() + } + result.Status = judge.StatusFail + result.Configuration = configName + return *result + case judge.StatusSkip: + result.Status = judge.StatusSkip + result.Configuration = configName + return *result + } + + // All turns completed (or completed + skipped with on_fail=skip_remaining + // that still produced judgeable data) — proceed to judge evaluation. + turnsTotal := len(caseCfg.Input.Turns) + return e.evaluateCaseSession(ctx, rt, caseCfg, configName, judgeCfg, turnsTotal, runAgent, aggregateSession, result) +} diff --git a/internal/evaluator/multiturn_test.go b/internal/evaluator/multiturn_test.go new file mode 100644 index 00000000..d3f9238a --- /dev/null +++ b/internal/evaluator/multiturn_test.go @@ -0,0 +1,976 @@ +package evaluator + +import ( + "context" + "errors" + "fmt" + "regexp" + "testing" + + "github.com/alibaba/skill-up/internal/agent" + "github.com/alibaba/skill-up/internal/config" + "github.com/alibaba/skill-up/internal/judge" + "github.com/alibaba/skill-up/internal/runtime" + "github.com/alibaba/skill-up/pkg/transcript" +) + +// mockResumerAgent implements both agent.Agent and agent.SessionResumer. +type mockResumerAgent struct { + mockAgent + + runTurnFunc func(ctx context.Context, rt runtime.Runtime, opts agent.ExecOptions, msg transcript.Message, sessionID string) (*agent.SessionResult, error) + turnCall int +} + +func (m *mockResumerAgent) RunTurn(ctx context.Context, rt runtime.Runtime, opts agent.ExecOptions, msg transcript.Message, sessionID string) (*agent.SessionResult, error) { + m.turnCall++ + if m.runTurnFunc != nil { + return m.runTurnFunc(ctx, rt, opts, msg, sessionID) + } + return &agent.SessionResult{ + FinalMessage: fmt.Sprintf("response to turn %d", msg.Turn), + SessionID: "test-session-123", + Turns: 1, + }, nil +} + +func TestSubstituteTemplate(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + content string + vars map[string]string + want string + wantErr bool + }{ + { + name: "no_placeholders", + content: "plain text without variables", + vars: map[string]string{"foo": "bar"}, + want: "plain text without variables", + }, + { + name: "single_substitution", + content: "The file is {{filename}}", + vars: map[string]string{"filename": "test.go"}, + want: "The file is test.go", + }, + { + name: "multiple_substitutions", + content: "{{greeting}} {{name}}, welcome to {{place}}", + vars: map[string]string{"greeting": "Hello", "name": "World", "place": "Go"}, + want: "Hello World, welcome to Go", + }, + { + name: "unresolved_variable", + content: "Use {{undefined_var}} here", + vars: map[string]string{"other": "value"}, + wantErr: true, + }, + { + name: "empty_vars_with_placeholder", + content: "{{missing}}", + vars: nil, + wantErr: true, + }, + { + name: "empty_content", + content: "", + vars: map[string]string{"x": "y"}, + want: "", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + got, err := substituteTemplate(tt.content, tt.vars) + if tt.wantErr { + if err == nil { + t.Fatalf("expected error, got result %q", got) + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != tt.want { + t.Fatalf("got %q, want %q", got, tt.want) + } + }) + } +} + +func TestEvaluatePostCondition(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + pc *config.PostCondition + response string + wantFail bool + }{ + { + name: "nil_post_condition", + pc: nil, + response: "anything", + wantFail: false, + }, + { + name: "must_contain_all_pass", + pc: &config.PostCondition{MustContainAll: []string{"foo", "bar"}}, + response: "foo and bar are here", + wantFail: false, + }, + { + name: "must_contain_all_fail", + pc: &config.PostCondition{MustContainAll: []string{"foo", "missing"}}, + response: "only foo is here", + wantFail: true, + }, + { + name: "must_contain_any_pass", + pc: &config.PostCondition{MustContainAny: []string{"alpha", "beta"}}, + response: "beta is present", + wantFail: false, + }, + { + name: "must_contain_any_fail", + pc: &config.PostCondition{MustContainAny: []string{"alpha", "beta"}}, + response: "neither is present", + wantFail: true, + }, + { + name: "must_not_contain_pass", + pc: &config.PostCondition{MustNotContain: []string{"error", "fail"}}, + response: "everything is fine", + wantFail: false, + }, + { + name: "must_not_contain_fail", + pc: &config.PostCondition{MustNotContain: []string{"error", "fail"}}, + response: "there was an error", + wantFail: true, + }, + { + name: "combined_rules_pass", + pc: &config.PostCondition{ + MustContainAll: []string{"success"}, + MustNotContain: []string{"error"}, + }, + response: "operation success", + wantFail: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + reason := evaluatePostCondition(tt.pc, tt.response) + if tt.wantFail && reason == "" { + t.Fatal("expected failure reason, got empty") + } + if !tt.wantFail && reason != "" { + t.Fatalf("unexpected failure: %s", reason) + } + }) + } +} + +func TestEvaluatePostCondition_CaseSensitive(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + pc *config.PostCondition + response string + wantFail bool + }{ + { + name: "must_contain_all_case_mismatch", + pc: &config.PostCondition{MustContainAll: []string{"Success"}}, + response: "success is here", // lowercase doesn't match uppercase + wantFail: true, + }, + { + name: "must_not_contain_case_mismatch_passes", + pc: &config.PostCondition{MustNotContain: []string{"error"}}, + response: "Error occurred", // different case, should pass + wantFail: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + reason := evaluatePostCondition(tt.pc, tt.response) + if tt.wantFail && reason == "" { + t.Fatal("expected failure reason, got empty") + } + if !tt.wantFail && reason != "" { + t.Fatalf("unexpected failure: %s", reason) + } + }) + } +} + +func TestCaptureVariables(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + rules []config.CaptureRule + response string + want map[string]string + wantErr bool + }{ + { + name: "named_group_value", + rules: []config.CaptureRule{{Variable: "id", Pattern: `ID: (?P\d+)`}}, + response: "Your ID: 42 is confirmed", + want: map[string]string{"id": "42"}, + }, + { + name: "first_capture_group", + rules: []config.CaptureRule{{Variable: "version", Pattern: `v(\d+\.\d+\.\d+)`}}, + response: "Version is v1.2.3", + want: map[string]string{"version": "1.2.3"}, + }, + { + name: "multiple_captures", + rules: []config.CaptureRule{{Variable: "a", Pattern: `a=(\w+)`}, {Variable: "b", Pattern: `b=(\w+)`}}, + response: "a=hello b=world", + want: map[string]string{"a": "hello", "b": "world"}, + }, + { + name: "no_match", + rules: []config.CaptureRule{{Variable: "x", Pattern: `xyz(\d+)`}}, + response: "no numbers here", + wantErr: true, + }, + { + name: "empty_pattern", + rules: []config.CaptureRule{{Variable: "x", Pattern: ""}}, + response: "anything", + wantErr: true, + }, + { + name: "invalid_regex", + rules: []config.CaptureRule{{Variable: "x", Pattern: `(?P<`}}, + response: "anything", + wantErr: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + got, err := captureVariables(tt.rules, tt.response) + if tt.wantErr { + if err == nil { + t.Fatalf("expected error, got %v", got) + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + for k, v := range tt.want { + if got[k] != v { + t.Fatalf("variable %q = %q, want %q", k, got[k], v) + } + } + }) + } +} + +func TestCaptureVariablesJSONPath(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + rules []config.CaptureRule + response string + want map[string]string + wantErr bool + }{ + { + name: "dot_path", + rules: []config.CaptureRule{{Variable: "name", JSONPath: "$.result.name"}}, + response: `{"result":{"name":"report.txt"}}`, + want: map[string]string{"name": "report.txt"}, + }, + { + name: "array_index", + rules: []config.CaptureRule{{Variable: "id", JSONPath: "$.items[1].id"}}, + response: `{"items":[{"id":"first"},{"id":42}]}`, + want: map[string]string{"id": "42"}, + }, + { + name: "missing_key", + rules: []config.CaptureRule{{Variable: "x", JSONPath: "$.missing"}}, + response: `{"present":true}`, + wantErr: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + got, err := captureVariables(tt.rules, tt.response) + if tt.wantErr { + if err == nil { + t.Fatalf("expected error, got %v", got) + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + for k, v := range tt.want { + if got[k] != v { + t.Fatalf("variable %q = %q, want %q", k, got[k], v) + } + } + }) + } +} + +func TestCaptureVariables_EdgeCases(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + rules []config.CaptureRule + response string + want map[string]string + wantErr bool + }{ + { + name: "multiple_unnamed_groups_error", + rules: []config.CaptureRule{{Variable: "x", Pattern: `(\w+)-(\d+)`}}, + response: "abc-123", + wantErr: true, // ambiguous: multiple unnamed groups without (?P...) + }, + { + name: "no_capture_groups_error", + rules: []config.CaptureRule{{Variable: "x", Pattern: `\d+`}}, + response: "42", + wantErr: true, // pattern matched but has no capture groups + }, + { + name: "named_group_plus_unnamed_uses_named", + rules: []config.CaptureRule{{Variable: "val", Pattern: `key=(\w+)-(?P\d+)`}}, + response: "key=abc-789", + want: map[string]string{"val": "789"}, // named group takes precedence + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + got, err := captureVariables(tt.rules, tt.response) + if tt.wantErr { + if err == nil { + t.Fatalf("expected error, got %v", got) + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + for k, v := range tt.want { + if got[k] != v { + t.Fatalf("variable %q = %q, want %q", k, got[k], v) + } + } + }) + } +} + +func TestMultiTurnStatus(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + results []TurnResult + want judge.Status + }{ + { + name: "all_completed", + results: []TurnResult{{Status: TurnCompleted}, {Status: TurnCompleted}}, + want: "", // proceed to judge + }, + { + name: "one_error", + results: []TurnResult{{Status: TurnCompleted}, {Status: TurnError}}, + want: judge.StatusError, + }, + { + name: "one_failed", + results: []TurnResult{{Status: TurnCompleted}, {Status: TurnFailed}}, + want: judge.StatusFail, + }, + { + name: "completed_then_skipped", + results: []TurnResult{{Status: TurnCompleted}, {Status: TurnSkipped}}, + want: "", // has completed turns, proceed to judge + }, + { + name: "all_skipped", + results: []TurnResult{{Status: TurnSkipped}, {Status: TurnSkipped}}, + want: judge.StatusSkip, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + got := multiTurnStatus(tt.results) + if got != tt.want { + t.Fatalf("got %q, want %q", got, tt.want) + } + }) + } +} + +func TestExecuteMultiTurn_HappyPath(t *testing.T) { + t.Parallel() + + workspace := t.TempDir() + rt := &mockRuntime{workspace: workspace} + + turnResponses := []string{"first response", "second response", "third response"} + turnIndex := 0 + + ag := &mockResumerAgent{ + mockAgent: mockAgent{name: "test-resumer"}, + runTurnFunc: func(_ context.Context, _ runtime.Runtime, _ agent.ExecOptions, msg transcript.Message, sessionID string) (*agent.SessionResult, error) { + idx := turnIndex + turnIndex++ + return &agent.SessionResult{ + FinalMessage: turnResponses[idx], + SessionID: "session-abc", + Turns: 1, + InputTokens: 10, + OutputTokens: 20, + }, nil + }, + } + + caseCfg := &config.CaseConfig{ + ID: "happy-multi", + Input: config.Input{ + Turns: []config.Turn{ + {Role: "user", Content: "first message"}, + {Role: "user", Content: "second message"}, + {Role: "user", Content: "third message"}, + }, + }, + } + + e := newTestEvaluator(EvalOptions{Agent: ag, EvalCfg: &config.EvalConfig{}}) + turnResults, aggResult, err := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(turnResults) != 3 { + t.Fatalf("got %d turn results, want 3", len(turnResults)) + } + for i, tr := range turnResults { + if tr.Status != TurnCompleted { + t.Fatalf("turn %d status = %v, want completed", i+1, tr.Status) + } + if tr.Response != turnResponses[i] { + t.Fatalf("turn %d response = %q, want %q", i+1, tr.Response, turnResponses[i]) + } + } + if aggResult == nil { + t.Fatal("aggregate result is nil") + return // staticcheck: unreachable but signals nil-safety to SA5011 + } + if got := aggResult.InputTokens; got != 30 { + t.Fatalf("aggregate input tokens = %d, want 30", got) + } + if got := aggResult.OutputTokens; got != 60 { + t.Fatalf("aggregate output tokens = %d, want 60", got) + } + if got := aggResult.Turns; got != 3 { + t.Fatalf("aggregate turns = %d, want 3", got) + } + if got := aggResult.FinalMessage; got != "third response" { + t.Fatalf("aggregate final message = %q, want %q", got, "third response") + } +} + +func TestExecuteMultiTurn_NormalizesCumulativeSessionResults(t *testing.T) { + t.Parallel() + + workspace := t.TempDir() + rt := &mockRuntime{workspace: workspace} + + cumulative := []*agent.SessionResult{ + { + FinalMessage: "first response", + SessionID: "session-abc", + Turns: 1, + InputTokens: 10, + OutputTokens: 4, + Transcript: transcript.Transcript{ + {Role: transcript.RoleUser, Content: "first", Turn: 1}, + {Role: transcript.RoleToolCall, ToolCall: &transcript.ToolCallInfo{Name: "first_tool"}, Turn: 1}, + {Role: transcript.RoleAssistant, Content: "first response", Turn: 1}, + }, + }, + { + FinalMessage: "second response", + SessionID: "session-abc", + Turns: 2, + InputTokens: 25, + OutputTokens: 9, + Transcript: transcript.Transcript{ + {Role: transcript.RoleUser, Content: "first", Turn: 1}, + {Role: transcript.RoleToolCall, ToolCall: &transcript.ToolCallInfo{Name: "first_tool"}, Turn: 1}, + {Role: transcript.RoleAssistant, Content: "first response", Turn: 1}, + {Role: transcript.RoleUser, Content: "second", Turn: 2}, + {Role: transcript.RoleToolCall, ToolCall: &transcript.ToolCallInfo{Name: "second_tool"}, Turn: 2}, + {Role: transcript.RoleAssistant, Content: "second response", Turn: 2}, + }, + }, + } + call := 0 + ag := &mockResumerAgent{ + mockAgent: mockAgent{name: "test-resumer"}, + runTurnFunc: func(_ context.Context, _ runtime.Runtime, _ agent.ExecOptions, _ transcript.Message, _ string) (*agent.SessionResult, error) { + result := cumulative[call] + call++ + return result, nil + }, + } + + caseCfg := &config.CaseConfig{ + ID: "cumulative", + Input: config.Input{ + Turns: []config.Turn{ + {Role: "user", Content: "first"}, + {Role: "user", Content: "second"}, + }, + }, + } + + e := newTestEvaluator(EvalOptions{Agent: ag, EvalCfg: &config.EvalConfig{}}) + turnResults, aggResult, err := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got := aggResult.InputTokens; got != 25 { + t.Fatalf("aggregate input tokens = %d, want cumulative high-water 25", got) + } + if got := aggResult.OutputTokens; got != 9 { + t.Fatalf("aggregate output tokens = %d, want cumulative high-water 9", got) + } + if got := len(aggResult.Transcript); got != 6 { + t.Fatalf("aggregate transcript length = %d, want 6 without duplicate turn 1", got) + } + if got := len(turnResults[1].Transcript); got != 3 { + t.Fatalf("turn 2 transcript length = %d, want only turn 2 messages", got) + } + calls := turnResults[1].Transcript.ToolCalls() + if len(calls) != 1 || calls[0].ToolCall.Name != "second_tool" { + t.Fatalf("turn 2 tool calls = %#v, want only second_tool", calls) + } +} + +func TestExecuteMultiTurn_PostConditionFail(t *testing.T) { + t.Parallel() + + workspace := t.TempDir() + rt := &mockRuntime{workspace: workspace} + + ag := &mockResumerAgent{ + mockAgent: mockAgent{name: "test-resumer"}, + runTurnFunc: func(_ context.Context, _ runtime.Runtime, _ agent.ExecOptions, msg transcript.Message, _ string) (*agent.SessionResult, error) { + return &agent.SessionResult{ + FinalMessage: "no keyword here", + SessionID: "s1", + Turns: 1, + }, nil + }, + } + + caseCfg := &config.CaseConfig{ + ID: "post-cond-fail", + Input: config.Input{ + Turns: []config.Turn{ + {Role: "user", Content: "turn one", PostCondition: &config.PostCondition{ + MustContainAll: []string{"expected_keyword"}, + OnFail: "fail", + }}, + {Role: "user", Content: "turn two"}, + }, + }, + } + + e := newTestEvaluator(EvalOptions{Agent: ag, EvalCfg: &config.EvalConfig{}}) + turnResults, _, err := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(turnResults) != 1 { + t.Fatalf("got %d turn results, want 1 (aborted at first turn)", len(turnResults)) + } + if turnResults[0].Status != TurnFailed { + t.Fatalf("turn 1 status = %v, want failed", turnResults[0].Status) + } +} + +func TestExecuteMultiTurn_PostConditionSkipRemaining(t *testing.T) { + t.Parallel() + + workspace := t.TempDir() + rt := &mockRuntime{workspace: workspace} + + ag := &mockResumerAgent{ + mockAgent: mockAgent{name: "test-resumer"}, + runTurnFunc: func(_ context.Context, _ runtime.Runtime, _ agent.ExecOptions, _ transcript.Message, _ string) (*agent.SessionResult, error) { + return &agent.SessionResult{ + FinalMessage: "partial response", + SessionID: "s2", + Turns: 1, + }, nil + }, + } + + caseCfg := &config.CaseConfig{ + ID: "skip-remaining", + Input: config.Input{ + Turns: []config.Turn{ + {Role: "user", Content: "first", PostCondition: &config.PostCondition{ + MustContainAll: []string{"required"}, + OnFail: "skip_remaining", + }}, + {Role: "user", Content: "second"}, + {Role: "user", Content: "third"}, + }, + }, + } + + e := newTestEvaluator(EvalOptions{Agent: ag, EvalCfg: &config.EvalConfig{}}) + turnResults, aggResult, err := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(turnResults) != 3 { + t.Fatalf("got %d turn results, want 3 (1 failed + 2 skipped)", len(turnResults)) + } + if turnResults[0].Status != TurnFailed { + t.Fatalf("turn 1 status = %v, want failed", turnResults[0].Status) + } + if turnResults[1].Status != TurnSkipped { + t.Fatalf("turn 2 status = %v, want skipped", turnResults[1].Status) + } + if turnResults[2].Status != TurnSkipped { + t.Fatalf("turn 3 status = %v, want skipped", turnResults[2].Status) + } + if aggResult == nil || aggResult.Turns != 1 { + t.Fatalf("aggregate executed turns = %v, want failed turn counted as executed", aggResult) + } +} + +func TestExecuteMultiTurn_TemplateSubstitution(t *testing.T) { + t.Parallel() + + workspace := t.TempDir() + rt := &mockRuntime{workspace: workspace} + + var receivedMessages []string + ag := &mockResumerAgent{ + mockAgent: mockAgent{name: "test-resumer"}, + runTurnFunc: func(_ context.Context, _ runtime.Runtime, _ agent.ExecOptions, msg transcript.Message, _ string) (*agent.SessionResult, error) { + receivedMessages = append(receivedMessages, msg.Content) + return &agent.SessionResult{ + FinalMessage: "file is report.txt version v2.0.0", + SessionID: "s3", + Turns: 1, + }, nil + }, + } + + caseCfg := &config.CaseConfig{ + ID: "template-sub", + Input: config.Input{ + Turns: []config.Turn{ + { + Role: "user", + Content: "Create a file", + Capture: []config.CaptureRule{ + {Variable: "filename", Pattern: `file is (\S+)`}, + {Variable: "ver", Pattern: `version (v\S+)`}, + }, + }, + {Role: "user", Content: "Now edit {{filename}} at {{ver}}"}, + }, + }, + } + + e := newTestEvaluator(EvalOptions{Agent: ag, EvalCfg: &config.EvalConfig{}}) + turnResults, _, err := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(turnResults) != 2 { + t.Fatalf("got %d turn results, want 2", len(turnResults)) + } + if len(receivedMessages) != 2 { + t.Fatalf("agent received %d messages, want 2", len(receivedMessages)) + } + expectedSecond := "Now edit report.txt at v2.0.0" + if receivedMessages[1] != expectedSecond { + t.Fatalf("second message = %q, want %q", receivedMessages[1], expectedSecond) + } + // Verify captured vars in turn results. + if turnResults[0].CapturedVars["filename"] != "report.txt" { + t.Fatalf("filename captured = %q, want report.txt", turnResults[0].CapturedVars["filename"]) + } +} + +func TestExecuteMultiTurn_AgentError(t *testing.T) { + t.Parallel() + + workspace := t.TempDir() + rt := &mockRuntime{workspace: workspace} + + callCount := 0 + ag := &mockResumerAgent{ + mockAgent: mockAgent{name: "test-resumer"}, + runTurnFunc: func(_ context.Context, _ runtime.Runtime, _ agent.ExecOptions, _ transcript.Message, _ string) (*agent.SessionResult, error) { + callCount++ + if callCount == 2 { + return nil, errors.New("agent crashed") + } + return &agent.SessionResult{FinalMessage: "ok", SessionID: "s4", Turns: 1}, nil + }, + } + + caseCfg := &config.CaseConfig{ + ID: "agent-error", + Input: config.Input{ + Turns: []config.Turn{ + {Role: "user", Content: "first"}, + {Role: "user", Content: "second"}, + {Role: "user", Content: "third"}, + }, + }, + } + + e := newTestEvaluator(EvalOptions{Agent: ag, EvalCfg: &config.EvalConfig{}}) + turnResults, _, err := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + + if err == nil { + t.Fatal("expected error, got nil") + } + if len(turnResults) != 2 { + t.Fatalf("got %d turn results, want 2 (1 completed + 1 error)", len(turnResults)) + } + if turnResults[0].Status != TurnCompleted { + t.Fatalf("turn 1 status = %v, want completed", turnResults[0].Status) + } + if turnResults[1].Status != TurnError { + t.Fatalf("turn 2 status = %v, want error", turnResults[1].Status) + } +} + +func TestExecuteMultiTurn_NoResumer(t *testing.T) { + t.Parallel() + + workspace := t.TempDir() + rt := &mockRuntime{workspace: workspace} + + // Plain mockAgent does NOT implement SessionResumer. + ag := &mockAgent{name: "no-resumer"} + + caseCfg := &config.CaseConfig{ + ID: "no-resumer", + Input: config.Input{ + Turns: []config.Turn{ + {Role: "user", Content: "first"}, + {Role: "user", Content: "second"}, + }, + }, + } + + e := newTestEvaluator(EvalOptions{Agent: ag, EvalCfg: &config.EvalConfig{}}) + _, _, err := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + + if err == nil { + t.Fatal("expected error for non-resumer agent") + } + if got := err.Error(); !contains(got, "SessionResumer") { + t.Fatalf("error = %q, want mention of SessionResumer", got) + } +} + +func TestExecuteMultiTurn_UnresolvedTemplate(t *testing.T) { + t.Parallel() + + workspace := t.TempDir() + rt := &mockRuntime{workspace: workspace} + + ag := &mockResumerAgent{ + mockAgent: mockAgent{name: "test-resumer"}, + runTurnFunc: func(_ context.Context, _ runtime.Runtime, _ agent.ExecOptions, _ transcript.Message, _ string) (*agent.SessionResult, error) { + return &agent.SessionResult{FinalMessage: "ok", SessionID: "s5", Turns: 1}, nil + }, + } + + caseCfg := &config.CaseConfig{ + ID: "unresolved", + Input: config.Input{ + Turns: []config.Turn{ + {Role: "user", Content: "first"}, + {Role: "user", Content: "Use {{undefined_var}} here"}, + }, + }, + } + + e := newTestEvaluator(EvalOptions{Agent: ag, EvalCfg: &config.EvalConfig{}}) + turnResults, _, err := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + + if err == nil { + t.Fatal("expected error for unresolved template") + } + if len(turnResults) != 2 { + t.Fatalf("got %d turn results, want 2 (1 completed + 1 error)", len(turnResults)) + } + if turnResults[1].Status != TurnError { + t.Fatalf("turn 2 status = %v, want error", turnResults[1].Status) + } +} + +func contains(s, substr string) bool { + return len(s) >= len(substr) && (s == substr || len(s) > 0 && containsCheck(s, substr)) +} + +func containsCheck(s, substr string) bool { + for i := 0; i <= len(s)-len(substr); i++ { + if s[i:i+len(substr)] == substr { + return true + } + } + return false +} + +func TestExtractCaptureValue(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + pattern string + input string + want string + wantErr bool + }{ + { + name: "named_group_value", + pattern: `ID: (?P\d+)`, + input: "ID: 42", + want: "42", + }, + { + name: "single_unnamed_group", + pattern: `ver=(\d+)`, + input: "ver=7", + want: "7", + }, + { + name: "multiple_unnamed_groups", + pattern: `(\w+)-(\d+)`, + input: "abc-123", + wantErr: true, + }, + { + name: "no_groups", + pattern: `\d+`, + input: "42", + wantErr: true, + }, + { + name: "named_plus_unnamed_prefers_named", + pattern: `(\w+)-(?P\d+)`, + input: "abc-99", + want: "99", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + re := regexp.MustCompile(tt.pattern) + match := re.FindStringSubmatch(tt.input) + if match == nil { + t.Fatal("pattern did not match input") + } + got, err := extractCaptureValue(re, match) + if tt.wantErr { + if err == nil { + t.Fatalf("expected error, got %q", got) + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != tt.want { + t.Fatalf("got %q, want %q", got, tt.want) + } + }) + } +} + +func TestCaptureVariableScoping(t *testing.T) { + t.Parallel() + + workspace := t.TempDir() + rt := &mockRuntime{workspace: workspace} + + callNum := 0 + ag := &mockResumerAgent{ + mockAgent: mockAgent{name: "test-resumer"}, + runTurnFunc: func(_ context.Context, _ runtime.Runtime, _ agent.ExecOptions, _ transcript.Message, _ string) (*agent.SessionResult, error) { + callNum++ + return &agent.SessionResult{FinalMessage: fmt.Sprintf("val=%d", callNum), SessionID: "s", Turns: 1}, nil + }, + } + + caseCfg := &config.CaseConfig{ + ID: "scope-test", + Input: config.Input{ + Turns: []config.Turn{ + {Role: "user", Content: "go", Capture: []config.CaptureRule{{Variable: "x", Pattern: `val=(\d+)`}}}, + {Role: "user", Content: "got {{x}}"}, + }, + }, + } + + e := newTestEvaluator(EvalOptions{Agent: ag, EvalCfg: &config.EvalConfig{}}) + + // First execution captures x=1. + results1, _, err1 := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + if err1 != nil { + t.Fatalf("first run error: %v", err1) + } + if results1[0].CapturedVars["x"] != "1" { + t.Fatalf("first run x = %q, want 1", results1[0].CapturedVars["x"]) + } + + // Second execution starts fresh — x should be captured as a new value (3), + // not carry over from first run. + results2, _, err2 := e.executeMultiTurn(context.Background(), rt, caseCfg, ag, agent.ExecOptions{}) + if err2 != nil { + t.Fatalf("second run error: %v", err2) + } + if results2[0].CapturedVars["x"] != "3" { + t.Fatalf("second run x = %q, want 3 (fresh scope)", results2[0].CapturedVars["x"]) + } +} diff --git a/internal/judge/agent_judge.go b/internal/judge/agent_judge.go index 7f7cf0f1..32ff9cec 100644 --- a/internal/judge/agent_judge.go +++ b/internal/judge/agent_judge.go @@ -70,14 +70,21 @@ type AgentJudge struct { // TimeoutSeconds bounds a single Evaluate call. <=0 means no judge-level // deadline; the parent context still applies. TimeoutSeconds int + + // JudgeSkills are installed judge Skills that must guide grading. + JudgeSkills []SkillInfo } // NewAgentJudge creates an AgentJudge with sensible defaults. -func NewAgentJudge(ag agent.Agent, rt runtime.Runtime, model string, criteria []string, passThreshold *float64, timeoutSeconds int) *AgentJudge { +func NewAgentJudge(ag agent.Agent, rt runtime.Runtime, model string, criteria []string, passThreshold *float64, timeoutSeconds int, judgeSkills ...[]SkillInfo) *AgentJudge { threshold := DefaultPassThreshold if passThreshold != nil { threshold = *passThreshold } + var skills []SkillInfo + if len(judgeSkills) > 0 { + skills = judgeSkills[0] + } return &AgentJudge{ Agent: ag, Runtime: rt, @@ -85,6 +92,7 @@ func NewAgentJudge(ag agent.Agent, rt runtime.Runtime, model string, criteria [] Criteria: criteria, PassThreshold: threshold, TimeoutSeconds: timeoutSeconds, + JudgeSkills: skills, } } @@ -102,7 +110,7 @@ func (j *AgentJudge) Evaluate(ctx context.Context, in Input) (*Result, error) { } // Build the judge prompt. - prompt := buildJudgePrompt(ctx, j.Criteria, in.FinalMessage, in.WorkspaceDiff, in.Transcript) + prompt := buildJudgePromptWithSkills(ctx, j.Criteria, in.FinalMessage, in.WorkspaceDiff, in.Transcript, j.JudgeSkills) messages := []transcript.Message{{Role: transcript.RoleUser, Content: prompt, Turn: 1}} // Get criterion results via agent.Agent. Snapshot parentCtx.Err() the @@ -398,6 +406,10 @@ func canRecoverAgentJudgeResult(err error, sessionResult *agent.SessionResult) b // buildJudgePrompt constructs the system + user prompt for the judge agent. func buildJudgePrompt(ctx context.Context, criteria []string, finalMessage, workspaceDiff string, transcriptData transcript.Transcript) string { + return buildJudgePromptWithSkills(ctx, criteria, finalMessage, workspaceDiff, transcriptData, nil) +} + +func buildJudgePromptWithSkills(ctx context.Context, criteria []string, finalMessage, workspaceDiff string, transcriptData transcript.Transcript, judgeSkills []SkillInfo) string { var sb strings.Builder sb.WriteString("You are an expert evaluator for an AI agent skill evaluation.\n") @@ -409,6 +421,22 @@ func buildJudgePrompt(ctx context.Context, criteria []string, finalMessage, work sb.WriteString("IMPORTANT: Your response MUST be valid JSON. If any string value contains double quotes, ") sb.WriteString("you MUST escape them with a backslash (e.g. \\\"example\\\"). Do NOT use unescaped double quotes inside string values.\n\n") + if len(judgeSkills) > 0 { + sb.WriteString("## Mandatory Judge Skill Use\n") + sb.WriteString("You MUST use the installed judge Skill(s) listed below as the authoritative grading rubric before evaluating the case. ") + sb.WriteString("Do not grade this case using only the inline criteria. The inline criteria identify result dimensions, while the judge Skill(s) define detailed rubric, constraints, and evidence rules. ") + sb.WriteString("If an inline criterion conflicts with a judge Skill, follow the judge Skill unless the criterion defines a more specific case-level acceptance condition.\n\n") + sb.WriteString("Installed judge Skill(s):\n") + for _, skill := range judgeSkills { + fmt.Fprintf(&sb, "- %s", skillIdentifier(skill)) + if skill.Target != "" { + fmt.Fprintf(&sb, " (target: %s)", skill.Target) + } + sb.WriteString("\n") + } + sb.WriteString("\n") + } + sb.WriteString("## Criteria\n") for i, c := range criteria { fmt.Fprintf(&sb, "%d. %s\n", i+1, c) @@ -455,3 +483,13 @@ func buildJudgePrompt(ctx context.Context, criteria []string, finalMessage, work return sb.String() } + +func skillIdentifier(skill SkillInfo) string { + if skill.Path != "" { + return skill.Path + } + if skill.Name != "" { + return skill.Name + } + return skill.Source +} diff --git a/internal/judge/agent_judge_test.go b/internal/judge/agent_judge_test.go index 2799dae4..842f67ed 100644 --- a/internal/judge/agent_judge_test.go +++ b/internal/judge/agent_judge_test.go @@ -344,6 +344,63 @@ func TestBuildJudgePrompt_ContainsAllParts(t *testing.T) { } } +func TestBuildJudgePromptWithSkills_RequiresJudgeSkillUse(t *testing.T) { + prompt := buildJudgePromptWithSkills( + context.Background(), + []string{"criterion A"}, + "Agent final message", + "", + nil, + []SkillInfo{ + {Source: "local_path", Path: "evals/fixtures/judge-skill", Target: "~/.claude/skills/judge-skill", Name: "judge-skill"}, + {Source: "local_path", Path: "evals/fixtures/security-judge", Name: "security-judge"}, + }, + ) + + checks := []string{ + "## Mandatory Judge Skill Use", + "You MUST use the installed judge Skill(s)", + "evals/fixtures/judge-skill", + "target: ~/.claude/skills/judge-skill", + "evals/fixtures/security-judge", + } + for _, c := range checks { + if !strings.Contains(prompt, c) { + t.Fatalf("prompt missing %q:\n%s", c, prompt) + } + } + if strings.Contains(prompt, "SKILL.md") || strings.Contains(prompt, "references/") { + t.Fatalf("prompt should list identifiers only, got:\n%s", prompt) + } +} + +func TestAgentJudge_EvaluatePassesJudgeSkillsInPrompt(t *testing.T) { + output := buildMockAgentOutput([]CriterionResult{ + {Criterion: "criterion A", Passed: true, Evidence: "used rubric"}, + }) + ag := &mockJudgeTestAgent{output: output} + rt := &mockJudgeTestRuntime{} + + j := NewAgentJudge( + ag, + rt, + "test-model", + []string{"criterion A"}, + nil, + 0, + []SkillInfo{{Source: "local_path", Path: "evals/fixtures/judge-skill", Name: "judge-skill"}}, + ) + _, err := j.Evaluate(context.Background(), Input{FinalMessage: "test"}) + assertNoError(t, err) + + if len(ag.lastMessages) != 1 { + t.Fatalf("lastMessages length = %d, want 1", len(ag.lastMessages)) + } + if !strings.Contains(ag.lastMessages[0].Content, "evals/fixtures/judge-skill") { + t.Fatalf("judge prompt missing skill path:\n%s", ag.lastMessages[0].Content) + } +} + func TestBuildJudgePrompt_TranscriptMarshalFailure_LogsAndOmitsErrorFromPrompt(t *testing.T) { var prompt string captured := captureLogOutput(t, func() { diff --git a/internal/judge/factory.go b/internal/judge/factory.go index 79de5a94..d327b5d8 100644 --- a/internal/judge/factory.go +++ b/internal/judge/factory.go @@ -38,7 +38,7 @@ func NewJudge(cfg config.JudgeConfig, ag agent.Agent, rt runtime.Runtime) (Judge if ag == nil { return nil, errors.New("agent_judge requires an Agent") } - return NewAgentJudge(ag, rt, cfg.Model, cfg.Criteria, cfg.PassThreshold, derefInt(cfg.TimeoutSeconds)), nil + return NewAgentJudge(ag, rt, cfg.Model, cfg.Criteria, cfg.PassThreshold, derefInt(cfg.TimeoutSeconds), SkillInfosFromRefs(cfg.Skills)), nil case "": // No judge configured — return nil, caller should handle diff --git a/internal/judge/factory_test.go b/internal/judge/factory_test.go index 2654ed10..02de16f9 100644 --- a/internal/judge/factory_test.go +++ b/internal/judge/factory_test.go @@ -98,6 +98,26 @@ func TestNewJudge_AgentJudge(t *testing.T) { } } +func TestNewJudge_AgentJudge_PropagatesJudgeSkills(t *testing.T) { + cfg := config.JudgeConfig{ + Type: "agent_judge", + Model: "test-model", + Criteria: []string{"c1"}, + Skills: []config.SkillRef{{Source: "local_path", Path: "evals/fixtures/judge-skill", Target: "~/.claude/skills/judge-skill"}}, + } + j, err := NewJudge(cfg, &mockJudgeTestAgent{}, &mockJudgeTestRuntime{}) + if err != nil { + t.Fatalf("NewJudge() error = %v", err) + } + aj, ok := j.(*AgentJudge) + if !ok { + t.Fatalf("expected *AgentJudge, got %T", j) + } + if len(aj.JudgeSkills) != 1 || aj.JudgeSkills[0].Path != "evals/fixtures/judge-skill" { + t.Fatalf("JudgeSkills = %#v", aj.JudgeSkills) + } +} + func TestNewJudge_AgentJudge_CustomThreshold(t *testing.T) { threshold := 0.9 cfg := config.JudgeConfig{ @@ -195,6 +215,7 @@ func TestMergeJudgeConfig_CaseOverridesGlobal(t *testing.T) { Model: "global-model", PassThreshold: &globalThreshold, Criteria: []string{"global-c"}, + Skills: []config.SkillRef{{Source: "local_path", Path: "evals/fixtures/global-judge"}}, TimeoutSeconds: intPtr(60), } caseLevel := config.JudgeConfig{ @@ -214,6 +235,31 @@ func TestMergeJudgeConfig_CaseOverridesGlobal(t *testing.T) { if merged.TimeoutSeconds == nil || *merged.TimeoutSeconds != 60 { t.Errorf("expected timeout 60 from global, got %v", merged.TimeoutSeconds) } + if len(merged.Skills) != 0 { + t.Errorf("expected case override to drop global skills, got %#v", merged.Skills) + } +} + +func TestMergeJudgeConfig_CaseAgentJudgeOverridesSkills(t *testing.T) { + global := config.JudgeConfig{ + Type: "agent_judge", + Model: "global-model", + Criteria: []string{"global-c"}, + Skills: []config.SkillRef{{Source: "local_path", Path: "evals/fixtures/global-judge"}}, + } + caseLevel := config.JudgeConfig{ + Type: "agent_judge", + Criteria: []string{"case-c"}, + Skills: []config.SkillRef{{Source: "local_path", Path: "evals/fixtures/case-judge"}}, + } + + merged := MergeJudgeConfig(global, caseLevel) + if merged.Model != "global-model" { + t.Fatalf("expected inherited model, got %q", merged.Model) + } + if len(merged.Skills) != 1 || merged.Skills[0].Path != "evals/fixtures/case-judge" { + t.Fatalf("expected case skills only, got %#v", merged.Skills) + } } func TestMergeJudgeConfig_CaseTimeoutOverridesGlobal(t *testing.T) { diff --git a/internal/judge/helpers_test.go b/internal/judge/helpers_test.go index 53108f63..a95af171 100644 --- a/internal/judge/helpers_test.go +++ b/internal/judge/helpers_test.go @@ -78,6 +78,7 @@ type mockJudgeTestAgent struct { // TimeoutSeconds correctly. ok mirrors ctx.Deadline()'s second return. observedDeadline time.Time observedDeadlineOK bool + lastMessages []transcript.Message } func (m *mockJudgeTestAgent) Name() string { return "mock-judge-agent" } @@ -98,8 +99,9 @@ func (m *mockJudgeTestAgent) Check(_ context.Context, _ runtime.Runtime) error { func (m *mockJudgeTestAgent) CheckCredentials(_ context.Context) error { return nil } -func (m *mockJudgeTestAgent) Run(ctx context.Context, _ runtime.Runtime, _ agent.ExecOptions, _ []transcript.Message) (*agent.SessionResult, error) { +func (m *mockJudgeTestAgent) Run(ctx context.Context, _ runtime.Runtime, _ agent.ExecOptions, messages []transcript.Message) (*agent.SessionResult, error) { m.observedDeadline, m.observedDeadlineOK = ctx.Deadline() + m.lastMessages = messages if m.runDelay > 0 { select { case <-ctx.Done(): diff --git a/internal/judge/judge.go b/internal/judge/judge.go index b221bebd..75d852b5 100644 --- a/internal/judge/judge.go +++ b/internal/judge/judge.go @@ -32,6 +32,22 @@ type Judge interface { Evaluate(ctx context.Context, in Input) (*Result, error) } +// InputTurnResult holds the outcome of a single conversation turn, visible to judges. +type InputTurnResult struct { + // TurnNumber is the 1-based index of this turn. + TurnNumber int + // Content is the user message sent to the agent. + Content string + // Response is the assistant response text for this turn. + Response string + // Transcript is the per-turn interaction record. + Transcript transcript.Transcript + // Status is the turn outcome: "completed", "skipped", "failed", "error". + Status string + // Reason describes why the turn was skipped/failed/errored. + Reason string +} + // Input carries artifacts needed for grading. // // It is the unified data boundary between the execution layer and the evaluation layer. @@ -70,6 +86,10 @@ type Input struct { // SessionResult is the full engine output, available for advanced judges. SessionResult *agent.SessionResult + // TurnResults holds per-turn outcomes for multi-turn evaluations. + // Nil for single-turn cases. + TurnResults []InputTurnResult + // TurnsExecuted is the number of turns actually executed. TurnsExecuted int diff --git a/internal/judge/judge_skill.go b/internal/judge/judge_skill.go new file mode 100644 index 00000000..1d0b90f6 --- /dev/null +++ b/internal/judge/judge_skill.go @@ -0,0 +1,32 @@ +package judge + +import ( + "path/filepath" + + "github.com/alibaba/skill-up/internal/config" +) + +// SkillInfo describes a judge Skill configured for agent_judge. +type SkillInfo struct { + Source string `json:"source,omitempty"` + Path string `json:"path,omitempty"` + Target string `json:"target,omitempty"` + Name string `json:"name,omitempty"` +} + +// SkillInfosFromRefs converts configured Skill refs into report-safe metadata. +func SkillInfosFromRefs(refs []config.SkillRef) []SkillInfo { + if len(refs) == 0 { + return nil + } + infos := make([]SkillInfo, 0, len(refs)) + for _, ref := range refs { + infos = append(infos, SkillInfo{ + Source: ref.Source, + Path: ref.Path, + Target: ref.Target, + Name: filepath.Base(filepath.Clean(ref.Path)), + }) + } + return infos +} diff --git a/internal/judge/rule_based.go b/internal/judge/rule_based.go index b31bb131..467b15b6 100644 --- a/internal/judge/rule_based.go +++ b/internal/judge/rule_based.go @@ -84,6 +84,14 @@ func evaluateAssertion(rule config.Rule, in Input) AssertionResult { return evalFilesExist(rule.FilesExist, in.WorkspacePath) case len(rule.FilesNotExist) > 0: return evalFilesNotExist(rule.FilesNotExist, in.WorkspacePath) + case rule.TurnResponseContains != nil: + return evalTurnResponseContains(rule.TurnResponseContains, in.TurnResults) + case rule.TurnResponseNotContains != nil: + return evalTurnResponseNotContains(rule.TurnResponseNotContains, in.TurnResults) + case rule.ToolCalledInTurn != nil: + return evalToolCalledInTurn(rule.ToolCalledInTurn, in.TurnResults) + case rule.ToolNotCalledInTurn != nil: + return evalToolNotCalledInTurn(rule.ToolNotCalledInTurn, in.TurnResults) default: return AssertionResult{ Text: "unknown_rule", @@ -162,7 +170,7 @@ func evalOutputContains(rule *config.OutputContainsRule, finalMessage string) As return AssertionResult{ Text: desc, Passed: true, - Evidence: "all output_contains checks passed", + Evidence: fmt.Sprintf("output satisfies all contains checks (%s)", strings.Join(descParts, ", ")), } } @@ -296,3 +304,188 @@ func argsMatch(expected, actual map[string]any) bool { } return true } + +// --------------------------------------------------------------------------- +// Per-turn rule evaluators +// --------------------------------------------------------------------------- + +// lookupTurn retrieves the turn result for a given 1-based turn number. +// Returns (turn, ok). If the turn is missing or not completed, it returns a +// failing AssertionResult describing why. +func lookupTurn(turnNum int, turns []InputTurnResult, ruleName string) (*InputTurnResult, *AssertionResult) { + if turnNum >= 1 { + for i := range turns { + if turns[i].TurnNumber == turnNum { + tr := &turns[i] + if tr.Status != "completed" { + ar := AssertionResult{ + Text: fmt.Sprintf("%s[turn=%d]", ruleName, turnNum), + Passed: false, + Evidence: fmt.Sprintf("turn %d has status %q (reason: %s); assertion requires completed turn", turnNum, tr.Status, tr.Reason), + } + return nil, &ar + } + return tr, nil + } + } + } + ar := AssertionResult{ + Text: fmt.Sprintf("%s[turn=%d]", ruleName, turnNum), + Passed: false, + Evidence: fmt.Sprintf("turn %d does not exist (executed %d turns)", turnNum, len(turns)), + } + return nil, &ar +} + +// evalTurnResponseContains checks a specific turn's response for required keywords. +func evalTurnResponseContains(rule *config.TurnResponseContainsRule, turns []InputTurnResult) AssertionResult { + tr, failAR := lookupTurn(rule.Turn, turns, "turn_response_contains") + if failAR != nil { + return *failAR + } + + // Check contains_all. + var missing []string + for _, kw := range rule.ContainsAll { + if !strings.Contains(tr.Response, kw) { + missing = append(missing, kw) + } + } + if len(missing) > 0 { + return AssertionResult{ + Text: fmt.Sprintf("turn_response_contains[turn=%d].contains_all%v", rule.Turn, rule.ContainsAll), + Passed: false, + Evidence: fmt.Sprintf("turn %d response missing required keywords: %v (checked: %v)", rule.Turn, missing, rule.ContainsAll), + } + } + + // Check contains_any. + if len(rule.ContainsAny) > 0 { + found := false + for _, kw := range rule.ContainsAny { + if strings.Contains(tr.Response, kw) { + found = true + break + } + } + if !found { + return AssertionResult{ + Text: fmt.Sprintf("turn_response_contains[turn=%d].contains_any%v", rule.Turn, rule.ContainsAny), + Passed: false, + Evidence: fmt.Sprintf("turn %d response does not contain any of %v", rule.Turn, rule.ContainsAny), + } + } + } + + // Build descriptive evidence. + var parts []string + if len(rule.ContainsAll) > 0 { + parts = append(parts, fmt.Sprintf("contains_all:%v", rule.ContainsAll)) + } + if len(rule.ContainsAny) > 0 { + parts = append(parts, fmt.Sprintf("contains_any:%v", rule.ContainsAny)) + } + desc := strings.Join(parts, ", ") + + return AssertionResult{ + Text: fmt.Sprintf("turn_response_contains[turn=%d]{%s}", rule.Turn, desc), + Passed: true, + Evidence: fmt.Sprintf("turn %d response contains all required keywords (%s)", rule.Turn, desc), + } +} + +// evalTurnResponseNotContains checks a specific turn's response for forbidden keywords. +func evalTurnResponseNotContains(rule *config.TurnResponseNotContainsRule, turns []InputTurnResult) AssertionResult { + tr, failAR := lookupTurn(rule.Turn, turns, "turn_response_not_contains") + if failAR != nil { + return *failAR + } + + for _, kw := range rule.NotContains { + if strings.Contains(tr.Response, kw) { + return AssertionResult{ + Text: fmt.Sprintf("turn_response_not_contains[turn=%d]{not:%v}", rule.Turn, rule.NotContains), + Passed: false, + Evidence: fmt.Sprintf("turn %d response contains forbidden keyword %q (checked: %v)", rule.Turn, kw, rule.NotContains), + } + } + } + + return AssertionResult{ + Text: fmt.Sprintf("turn_response_not_contains[turn=%d]{not:%v}", rule.Turn, rule.NotContains), + Passed: true, + Evidence: fmt.Sprintf("turn %d response does not contain any of the forbidden keywords %v", rule.Turn, rule.NotContains), + } +} + +// evalToolCalledInTurn checks that a specific tool was called during a specific turn. +func evalToolCalledInTurn(rule *config.ToolCalledInTurnRule, turns []InputTurnResult) AssertionResult { + tr, failAR := lookupTurn(rule.Turn, turns, "tool_called_in_turn") + if failAR != nil { + return *failAR + } + + calls := tr.Transcript.ToolCalls() + for _, msg := range calls { + if msg.ToolCall == nil { + continue + } + if msg.ToolCall.Name != rule.Name { + continue + } + // Name matched; check args if specified. + if len(rule.Args) == 0 { + return AssertionResult{ + Text: fmt.Sprintf("tool_called_in_turn[turn=%d]: %s", rule.Turn, rule.Name), + Passed: true, + Evidence: fmt.Sprintf("tool %q was called in turn %d", rule.Name, rule.Turn), + } + } + if argsMatch(rule.Args, msg.ToolCall.Arguments) { + return AssertionResult{ + Text: fmt.Sprintf("tool_called_in_turn[turn=%d]: %s (with args)", rule.Turn, rule.Name), + Passed: true, + Evidence: fmt.Sprintf("tool %q was called in turn %d with matching args", rule.Name, rule.Turn), + } + } + } + + // Not found. + evidence := fmt.Sprintf("tool %q was not called in turn %d", rule.Name, rule.Turn) + if len(rule.Args) > 0 { + evidence = fmt.Sprintf("tool %q was not called in turn %d with matching args %v", rule.Name, rule.Turn, rule.Args) + } + return AssertionResult{ + Text: fmt.Sprintf("tool_called_in_turn[turn=%d]: %s", rule.Turn, rule.Name), + Passed: false, + Evidence: evidence, + } +} + +// evalToolNotCalledInTurn checks that a specific tool was NOT called during a specific turn. +func evalToolNotCalledInTurn(rule *config.ToolNotCalledInTurnRule, turns []InputTurnResult) AssertionResult { + tr, failAR := lookupTurn(rule.Turn, turns, "tool_not_called_in_turn") + if failAR != nil { + return *failAR + } + + calls := tr.Transcript.ToolCalls() + for _, msg := range calls { + if msg.ToolCall == nil { + continue + } + if msg.ToolCall.Name == rule.Name { + return AssertionResult{ + Text: fmt.Sprintf("tool_not_called_in_turn[turn=%d]: %s", rule.Turn, rule.Name), + Passed: false, + Evidence: fmt.Sprintf("tool %q was called in turn %d but should not have been", rule.Name, rule.Turn), + } + } + } + + return AssertionResult{ + Text: fmt.Sprintf("tool_not_called_in_turn[turn=%d]: %s", rule.Turn, rule.Name), + Passed: true, + Evidence: fmt.Sprintf("tool %q was not called in turn %d as expected", rule.Name, rule.Turn), + } +} diff --git a/internal/judge/rule_based_test.go b/internal/judge/rule_based_test.go index a7b66520..c00c0412 100644 --- a/internal/judge/rule_based_test.go +++ b/internal/judge/rule_based_test.go @@ -411,3 +411,324 @@ func TestArgsMatch_Empty(t *testing.T) { t.Fatal("empty expected should always match") } } + +// --------------------------------------------------------------------------- +// turn_response_contains +// --------------------------------------------------------------------------- + +func TestRuleBased_TurnResponseContains_ContainsAll_Pass(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "hello world", Status: "completed"}, + {TurnNumber: 2, Response: "I created file.go and test_file.go", Status: "completed"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseContains: &config.TurnResponseContainsRule{ + Turn: 2, + ContainsAll: []string{"file.go", "test_file.go"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 2, TurnsExecuted: 2}) + assertNoError(t, err) + assertStatus(t, r, StatusPass) +} + +func TestRuleBased_TurnResponseContains_ContainsAll_Fail(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "I created file.go only", Status: "completed"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseContains: &config.TurnResponseContainsRule{ + Turn: 1, + ContainsAll: []string{"file.go", "test_file.go"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusFail) + if r.AssertionResults[0].Evidence == "" { + t.Fatal("expected evidence with missing keywords") + } +} + +func TestRuleBased_TurnResponseContains_ContainsAny_Pass(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "result is success", Status: "completed"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseContains: &config.TurnResponseContainsRule{ + Turn: 1, + ContainsAny: []string{"success", "ok"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusPass) +} + +func TestRuleBased_TurnResponseContains_ContainsAny_Fail(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "something else", Status: "completed"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseContains: &config.TurnResponseContainsRule{ + Turn: 1, + ContainsAny: []string{"success", "ok"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusFail) +} + +func TestRuleBased_TurnResponseContains_MissingTurn(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "only one turn", Status: "completed"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseContains: &config.TurnResponseContainsRule{ + Turn: 3, + ContainsAll: []string{"something"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 3, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusFail) + if r.AssertionResults[0].Evidence == "" { + t.Fatal("expected evidence about missing turn") + } +} + +func TestRuleBased_TurnResponseContains_UsesTurnNumber(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 2, Response: "target turn response", Status: "completed"}, + {TurnNumber: 1, Response: "first turn response", Status: "completed"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseContains: &config.TurnResponseContainsRule{ + Turn: 2, + ContainsAll: []string{"target"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 2, TurnsExecuted: 2}) + assertNoError(t, err) + assertStatus(t, r, StatusPass) +} + +func TestRuleBased_TurnResponseContains_SkippedTurn(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "hello", Status: "completed"}, + {TurnNumber: 2, Response: "", Status: "skipped", Reason: "post_condition skip_remaining"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseContains: &config.TurnResponseContainsRule{ + Turn: 2, + ContainsAll: []string{"hello"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 2, TurnsExecuted: 2}) + assertNoError(t, err) + assertStatus(t, r, StatusFail) + if r.AssertionResults[0].Evidence == "" { + t.Fatal("expected evidence about skipped turn") + } +} + +// --------------------------------------------------------------------------- +// turn_response_not_contains +// --------------------------------------------------------------------------- + +func TestRuleBased_TurnResponseNotContains_PerTurn_Pass(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "clean output", Status: "completed"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseNotContains: &config.TurnResponseNotContainsRule{ + Turn: 1, + NotContains: []string{"error", "panic"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusPass) +} + +func TestRuleBased_TurnResponseNotContains_PerTurn_Fail(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "runtime error occurred", Status: "completed"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseNotContains: &config.TurnResponseNotContainsRule{ + Turn: 1, + NotContains: []string{"error", "panic"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusFail) +} + +func TestRuleBased_TurnResponseNotContains_MissingTurn(t *testing.T) { + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + TurnResponseNotContains: &config.TurnResponseNotContainsRule{ + Turn: 1, + NotContains: []string{"error"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: nil, TurnsTotal: 1, TurnsExecuted: 0}) + assertNoError(t, err) + assertStatus(t, r, StatusFail) +} + +// --------------------------------------------------------------------------- +// tool_called_in_turn +// --------------------------------------------------------------------------- + +func TestRuleBased_ToolCalledInTurn_Pass(t *testing.T) { + turns := []InputTurnResult{ + { + TurnNumber: 1, + Response: "done", + Status: "completed", + Transcript: transcript.Transcript{ + {Role: "tool_call", ToolCall: &transcript.ToolCallInfo{Name: "write_file", Arguments: map[string]any{"path": "main.go"}}}, + }, + }, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + ToolCalledInTurn: &config.ToolCalledInTurnRule{ + Turn: 1, + Name: "write_file", + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusPass) +} + +func TestRuleBased_ToolCalledInTurn_WithArgs_Pass(t *testing.T) { + turns := []InputTurnResult{ + { + TurnNumber: 1, + Response: "done", + Status: "completed", + Transcript: transcript.Transcript{ + {Role: "tool_call", ToolCall: &transcript.ToolCallInfo{Name: "write_file", Arguments: map[string]any{"path": "main.go", "content": "pkg"}}}, + }, + }, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + ToolCalledInTurn: &config.ToolCalledInTurnRule{ + Turn: 1, + Name: "write_file", + Args: map[string]any{"path": "main.go"}, + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusPass) +} + +func TestRuleBased_ToolCalledInTurn_Fail(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "done", Status: "completed", Transcript: transcript.Transcript{ + {Role: "tool_call", ToolCall: &transcript.ToolCallInfo{Name: "read_file"}}, + }}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + ToolCalledInTurn: &config.ToolCalledInTurnRule{Turn: 1, Name: "write_file"}, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusFail) +} + +func TestRuleBased_ToolCalledInTurn_FailedTurn(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "", Status: "failed", Reason: "post_condition failed"}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + ToolCalledInTurn: &config.ToolCalledInTurnRule{ + Turn: 1, + Name: "write_file", + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, StatusFail) +} + +// --------------------------------------------------------------------------- +// tool_not_called_in_turn +// --------------------------------------------------------------------------- + +func TestRuleBased_ToolNotCalledInTurn_PassAndFail(t *testing.T) { + tests := []struct { + name string + toolName string + ruleName string + wantStatus Status + }{ + {"pass_tool_absent", "read_file", "delete_file", StatusPass}, + {"fail_tool_present", "delete_file", "delete_file", StatusFail}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + turns := []InputTurnResult{ + {TurnNumber: 1, Response: "done", Status: "completed", Transcript: transcript.Transcript{ + {Role: "tool_call", ToolCall: &transcript.ToolCallInfo{Name: tt.toolName}}, + {Role: "assistant", Content: "result"}, + }}, + } + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + ToolNotCalledInTurn: &config.ToolNotCalledInTurnRule{Turn: 1, Name: tt.ruleName}, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: turns, TurnsTotal: 1, TurnsExecuted: 1}) + assertNoError(t, err) + assertStatus(t, r, tt.wantStatus) + }) + } +} + +func TestRuleBased_ToolNotCalledInTurn_MissingTurn(t *testing.T) { + j := NewRuleBasedJudge(config.JudgeConfig{ + Success: []config.Rule{{ + ToolNotCalledInTurn: &config.ToolNotCalledInTurnRule{ + Turn: 5, + Name: "delete_file", + }, + }}, + }) + r, err := j.Evaluate(context.Background(), Input{TurnResults: nil, TurnsTotal: 5, TurnsExecuted: 0}) + assertNoError(t, err) + assertStatus(t, r, StatusFail) +} diff --git a/internal/report/html.go b/internal/report/html.go index da0e77b8..37cc1c06 100644 --- a/internal/report/html.go +++ b/internal/report/html.go @@ -79,18 +79,29 @@ type embeddedSummary struct { } type embeddedCase struct { - ID string `json:"id"` - Title string `json:"title,omitempty"` - Status string `json:"status"` - DurationMs int64 `json:"duration_ms"` - Duration string `json:"duration"` - Turns int `json:"turns"` - Error string `json:"error,omitempty"` - Grading *embeddedGrading `json:"grading,omitempty"` - Configuration string `json:"configuration,omitempty"` - Prompt string `json:"prompt,omitempty"` - Response string `json:"response,omitempty"` - Baseline *embeddedCase `json:"baseline,omitempty"` + ID string `json:"id"` + Title string `json:"title,omitempty"` + Status string `json:"status"` + DurationMs int64 `json:"duration_ms"` + Duration string `json:"duration"` + Turns int `json:"turns"` + Error string `json:"error,omitempty"` + Grading *embeddedGrading `json:"grading,omitempty"` + Configuration string `json:"configuration,omitempty"` + Prompt string `json:"prompt,omitempty"` + Response string `json:"response,omitempty"` + Baseline *embeddedCase `json:"baseline,omitempty"` + TurnResults []embeddedTurn `json:"turn_results,omitempty"` + JudgeSkills []judge.SkillInfo `json:"judge_skills,omitempty"` +} + +// embeddedTurn holds per-turn data for the HTML report JavaScript. +type embeddedTurn struct { + TurnNumber int `json:"turn_number"` + Content string `json:"content"` + Response string `json:"response"` + Status string `json:"status"` + Reason string `json:"reason,omitempty"` } type embeddedGrading struct { @@ -135,6 +146,8 @@ func caseResultToEmbeddedCase(cr CaseResult) embeddedCase { Configuration: cr.Configuration, Prompt: cr.Prompt, Response: cr.Response, + TurnResults: caseTurnResultsToEmbedded(cr.TurnResults), + JudgeSkills: cr.JudgeSkills, } if cr.Grading != nil { eg := &embeddedGrading{ @@ -158,6 +171,17 @@ func caseResultToEmbeddedCase(cr CaseResult) embeddedCase { return ec } +func caseTurnResultsToEmbedded(turns []CaseTurnResult) []embeddedTurn { + if len(turns) == 0 { + return nil + } + out := make([]embeddedTurn, len(turns)) + for i, tr := range turns { + out[i] = embeddedTurn(tr) + } + return out +} + func groupCaseResults(results []CaseResult) (map[string]*caseGroup, []string) { grouped := make(map[string]*caseGroup) orderedIDs := make([]string, 0, len(results)) diff --git a/internal/report/junit.go b/internal/report/junit.go index 39273ba4..9382fc7f 100644 --- a/internal/report/junit.go +++ b/internal/report/junit.go @@ -5,6 +5,7 @@ import ( "encoding/xml" "fmt" "io" + "strconv" "strings" "time" @@ -63,13 +64,23 @@ type junitTestSuite struct { // junitTestCase is a element. type junitTestCase struct { - XMLName xml.Name `xml:"testcase"` - Name string `xml:"name,attr"` - ClassName string `xml:"classname,attr"` - Time string `xml:"time,attr"` - Failure *junitFailure `xml:"failure,omitempty"` - Error *junitError `xml:"error,omitempty"` - Skipped *junitSkipped `xml:"skipped,omitempty"` + XMLName xml.Name `xml:"testcase"` + Name string `xml:"name,attr"` + ClassName string `xml:"classname,attr"` + Time string `xml:"time,attr"` + Properties *junitProperties `xml:"properties,omitempty"` + Failure *junitFailure `xml:"failure,omitempty"` + Error *junitError `xml:"error,omitempty"` + Skipped *junitSkipped `xml:"skipped,omitempty"` +} + +type junitProperties struct { + Properties []junitProperty `xml:"property"` +} + +type junitProperty struct { + Name string `xml:"name,attr"` + Value string `xml:"value,attr"` } // junitFailure is a element. @@ -109,6 +120,7 @@ func (r *JUnitReporter) buildTestSuite(in Input) junitTestSuites { ClassName: in.SkillName, Time: fmt.Sprintf("%.3f", float64(cr.DurationMs)/1000.0), } + tc.Properties = buildJudgeSkillProperties(cr) switch cr.Status { case judge.StatusFail: @@ -157,7 +169,25 @@ func (r *JUnitReporter) buildTestSuite(in Input) junitTestSuites { return junitTestSuites{Suites: []junitTestSuite{suite}} } +func buildJudgeSkillProperties(cr CaseResult) *junitProperties { + props := []junitProperty{ + {Name: "judge.skills.count", Value: strconv.Itoa(len(cr.JudgeSkills))}, + } + for i, skill := range cr.JudgeSkills { + prefix := fmt.Sprintf("judge.skills.%d.", i) + props = append(props, + junitProperty{Name: prefix + "source", Value: skill.Source}, + junitProperty{Name: prefix + "path", Value: skill.Path}, + junitProperty{Name: prefix + "target", Value: skill.Target}, + junitProperty{Name: prefix + "name", Value: skill.Name}, + ) + } + return &junitProperties{Properties: props} +} + // buildFailureBody extracts failed assertion details for the body. +// Turn-scoped assertions already include turn numbers in their text field, +// making CI failure output directly actionable. func buildFailureBody(cr CaseResult) string { if cr.Grading == nil { return "" @@ -168,5 +198,13 @@ func buildFailureBody(cr CaseResult) string { lines = append(lines, fmt.Sprintf("- %s: %s", ar.Text, ar.Evidence)) } } + // Append turn summary when multi-turn results are present. + if len(cr.TurnResults) > 0 { + for _, tr := range cr.TurnResults { + if tr.Status != "completed" { + lines = append(lines, fmt.Sprintf("- turn %d: status=%s reason=%s", tr.TurnNumber, tr.Status, tr.Reason)) + } + } + } return strings.Join(lines, "\n") } diff --git a/internal/report/reporter.go b/internal/report/reporter.go index 275ad10e..4e1de879 100644 --- a/internal/report/reporter.go +++ b/internal/report/reporter.go @@ -55,18 +55,29 @@ func (in Input) OverallPassRate() float64 { // CaseResult represents the result of a single case execution. type CaseResult struct { - CaseID string `json:"case_id"` - Title string `json:"title"` - Status judge.Status `json:"status"` - DurationMs int64 `json:"duration_ms"` - Turns int `json:"turns"` - InputTokens int `json:"input_tokens"` - OutputTokens int `json:"output_tokens"` - Error string `json:"error,omitempty"` - Grading *judge.Result `json:"grading"` - Configuration string `json:"configuration,omitempty"` // "with_skill" or "without_skill" - Prompt string `json:"prompt,omitempty"` // input prompt sent to the agent - Response string `json:"response,omitempty"` // agent final message + CaseID string `json:"case_id"` + Title string `json:"title"` + Status judge.Status `json:"status"` + DurationMs int64 `json:"duration_ms"` + Turns int `json:"turns"` + InputTokens int `json:"input_tokens"` + OutputTokens int `json:"output_tokens"` + Error string `json:"error,omitempty"` + Grading *judge.Result `json:"grading"` + JudgeSkills []judge.SkillInfo `json:"judge_skills,omitempty"` + Configuration string `json:"configuration,omitempty"` // "with_skill" or "without_skill" + Prompt string `json:"prompt,omitempty"` // input prompt sent to the agent + Response string `json:"response,omitempty"` // agent final message + TurnResults []CaseTurnResult `json:"turn_results,omitempty"` // per-turn outcomes; nil for single-turn +} + +// CaseTurnResult holds the outcome of a single turn for reporting purposes. +type CaseTurnResult struct { + TurnNumber int `json:"turn_number"` + Content string `json:"content"` + Response string `json:"response"` + Status string `json:"status"` + Reason string `json:"reason,omitempty"` } // BenchmarkResult is the top-level structure for benchmark.json. diff --git a/internal/report/reporter_test.go b/internal/report/reporter_test.go index 276fb8f9..b8ad38c7 100644 --- a/internal/report/reporter_test.go +++ b/internal/report/reporter_test.go @@ -31,6 +31,9 @@ func sampleInput() Input { Status: judge.StatusPass, DurationMs: 45200, Turns: 5, + JudgeSkills: []judge.SkillInfo{ + {Source: "local_path", Path: "evals/fixtures/judge-skill", Target: "~/.claude/skills/judge-skill", Name: "judge-skill"}, + }, Grading: &judge.Result{ Status: judge.StatusPass, TurnsExecuted: 5, @@ -115,6 +118,9 @@ func TestJSONReporter_Write(t *testing.T) { if parsed.CaseResults[1].Status != judge.StatusFail { t.Fatalf("expected second case FAIL, got %s", parsed.CaseResults[1].Status) } + if len(parsed.CaseResults[0].JudgeSkills) != 1 || parsed.CaseResults[0].JudgeSkills[0].Path != "evals/fixtures/judge-skill" { + t.Fatalf("judge_skills not preserved in JSON: %#v", parsed.CaseResults[0].JudgeSkills) + } } func TestJSONReporter_ContainsAssertions(t *testing.T) { @@ -174,6 +180,12 @@ func TestJUnitReporter_Write(t *testing.T) { if !strings.Contains(content, `errors="1"`) { t.Fatal("expected 1 error") } + if !strings.Contains(content, `name="judge.skills.count" value="1"`) { + t.Fatal("junit should include judge skill count property") + } + if !strings.Contains(content, `name="judge.skills.0.path" value="evals/fixtures/judge-skill"`) { + t.Fatal("junit should include judge skill path property") + } } func TestJUnitReporter_FailureDetails(t *testing.T) { @@ -232,6 +244,9 @@ func TestHTMLReporter_Write(t *testing.T) { if !strings.Contains(content, "edge-case-null") { t.Fatal("missing failed case ID") } + if !strings.Contains(content, "evals/fixtures/judge-skill") { + t.Fatal("missing judge skill path in embedded report data") + } } func TestHTMLReporter_ContainsAssertionDetails(t *testing.T) { @@ -251,6 +266,40 @@ func TestHTMLReporter_ContainsAssertionDetails(t *testing.T) { } } +func TestHTMLReporter_SynthesizedTurnFailurePassRateScript(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "result.html") + r := &HTMLReporter{OutputPath: path} + + input := Input{ + SkillName: "multi-turn", + CaseResults: []CaseResult{{ + CaseID: "turn-failure", + Status: judge.StatusFail, + Turns: 2, + TurnResults: []CaseTurnResult{ + {TurnNumber: 1, Status: "completed", Response: "ok"}, + {TurnNumber: 2, Status: "failed", Reason: "missing token"}, + }, + }}, + } + if err := r.Write(context.Background(), input); err != nil { + t.Fatalf("HTMLReporter.Write failed: %v", err) + } + + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read html file: %v", err) + } + content := string(data) + if !strings.Contains(content, "Math.round((passed.length / total) * 100)") { + t.Fatal("HTML should compute synthesized turn-failure pass rate from passed/total") + } + if !strings.Contains(content, "total === 0 ? 0") { + t.Fatal("HTML should guard synthesized turn-failure pass rate when total is zero") + } +} + // --------------------------------------------------------------------------- // Input helpers // --------------------------------------------------------------------------- diff --git a/internal/report/templates/report.html b/internal/report/templates/report.html index 5ebf22c1..b6ed43c0 100644 --- a/internal/report/templates/report.html +++ b/internal/report/templates/report.html @@ -107,6 +107,22 @@ .prompt-box { background: #f8fafc; border: 1px solid var(--border); border-radius: 4px; padding: 0.75rem 1rem; font-size: 0.875rem; color: #555; margin-bottom: 0.5rem; } .prompt-box strong { color: var(--text); } .response-box { background: #f8f8f8; border: 1px solid #e8e8e8; border-radius: 6px; padding: 0.75rem 1rem; font-size: 0.85rem; white-space: pre-wrap; word-wrap: break-word; max-height: 300px; overflow-y: auto; } + .turn-card { border: 1px solid var(--border); border-radius: var(--radius); margin-bottom: 1rem; overflow: hidden; } + .turn-card:last-child { margin-bottom: 0; } + .turn-header { background: var(--bg); padding: 0.5rem 1rem; font-size: 0.75rem; font-weight: 600; color: var(--text-muted); display: flex; align-items: center; gap: 0.5rem; border-bottom: 1px solid var(--border); } + .turn-header .turn-badge { background: var(--accent); color: white; padding: 0.1rem 0.5rem; border-radius: 9999px; font-size: 0.65rem; } + .turn-header .turn-status { margin-left: auto; font-size: 0.7rem; font-weight: 500; } + .turn-header .turn-status.ts-completed { color: var(--green); } + .turn-header .turn-status.ts-skipped { color: var(--skip); } + .turn-header .turn-status.ts-failed { color: var(--red); } + .turn-header .turn-status.ts-error { color: var(--error-color); } + .turn-prompt { padding: 0.75rem 1rem; background: #f0f4ff; border-bottom: 1px solid var(--border); } + .turn-prompt-label { font-size: 0.7rem; font-weight: 600; color: var(--accent); text-transform: uppercase; letter-spacing: 0.04em; margin-bottom: 0.25rem; } + .turn-prompt-content { font-size: 0.85rem; color: var(--text); white-space: pre-wrap; word-wrap: break-word; } + .turn-response { padding: 0.75rem 1rem; background: var(--surface); } + .turn-response-label { font-size: 0.7rem; font-weight: 600; color: var(--green); text-transform: uppercase; letter-spacing: 0.04em; margin-bottom: 0.25rem; } + .turn-response-content { font-size: 0.85rem; color: var(--text); white-space: pre-wrap; word-wrap: break-word; max-height: 250px; overflow-y: auto; } + .turn-skipped-msg { padding: 0.75rem 1rem; background: var(--skip-bg); color: var(--skip); font-size: 0.8rem; font-style: italic; } .comparison { display: grid; grid-template-columns: 1fr 1fr; gap: 0; } .config-col { padding: 1rem; } .config-col:first-child { border-right: 1px solid var(--border); } @@ -345,8 +361,14 @@

Review Complete

document.getElementById("case-info").innerHTML = info; var promptSec = document.getElementById("prompt-section"); - if (c.prompt) { promptSec.style.display = "block"; document.getElementById("prompt-text").textContent = c.prompt; } - else { promptSec.style.display = "none"; } + if (c.turn_results && c.turn_results.length > 0) { + promptSec.style.display = "none"; + } else if (c.prompt) { + promptSec.style.display = "block"; + document.getElementById("prompt-text").textContent = c.prompt; + } else { + promptSec.style.display = "none"; + } var errSec = document.getElementById("error-section"); if (c.error) { errSec.style.display = "block"; document.getElementById("error-text").textContent = c.error; } @@ -372,8 +394,38 @@

Review Complete

function renderResponse(c) { var section = document.getElementById("response-section"); var body = document.getElementById("response-body"); + var header = section.querySelector(".section-header"); + + // Multi-turn: render full conversation with per-turn prompt and response. + if (c.turn_results && c.turn_results.length > 0) { + section.style.display = "block"; + header.textContent = "Conversation (" + c.turn_results.length + " turns)"; + var html = '
'; + for (var i = 0; i < c.turn_results.length; i++) { + var tr = c.turn_results[i]; + var statusCls = "ts-" + (tr.status || "completed"); + html += '
'; + html += '
Turn ' + tr.turn_number + ''; + html += '' + esc(tr.status || "completed") + '
'; + html += '
Prompt
'; + html += '
' + esc(tr.content || "") + '
'; + if (tr.status === "skipped") { + html += '
This turn was skipped' + (tr.reason ? ': ' + esc(tr.reason) : '') + '
'; + } else if (tr.response) { + html += '
Response
'; + html += '
' + esc(tr.response) + '
'; + } + html += '
'; + } + html += '
'; + body.innerHTML = html; + return; + } + + // Single-turn fallback. if (!c.response && !(c.baseline && c.baseline.response)) { section.style.display = "none"; return; } section.style.display = "block"; + header.textContent = "Response"; if (c.baseline) { var html = '
'; html += '

With Skill'; @@ -395,24 +447,60 @@

Review Complete

function renderGrades(c) { var section = document.getElementById("grades-section"); var content = document.getElementById("grades-content"); - if (!c.grading && !(c.baseline && c.baseline.grading)) { section.style.display = "none"; return; } + + // Check if we have grading data or can synthesize from turn failures. + var hasGrading = c.grading || (c.baseline && c.baseline.grading); + var hasTurnFailures = !hasGrading && c.turn_results && c.turn_results.some(function(tr) { return tr.status === "failed" || tr.status === "error"; }); + + if (!hasGrading && !hasTurnFailures) { section.style.display = "none"; return; } section.style.display = "block"; content.classList.remove("open"); document.getElementById("grades-arrow").classList.remove("open"); var html = '
'; - if (c.baseline) { - html += '
'; - html += '

With Skill

' + renderGradingBlock(c.grading) + '
'; - html += '

Without Skill

' + renderGradingBlock(c.baseline.grading) + '
'; - html += '
'; + if (hasGrading) { + if (c.baseline) { + html += '
'; + html += '

With Skill

' + renderGradingBlock(c.grading) + '
'; + html += '

Without Skill

' + renderGradingBlock(c.baseline.grading) + '
'; + html += '
'; + } else { + html += renderGradingBlock(c.grading); + } } else { - html += renderGradingBlock(c.grading); + // Synthesize grading display from turn post-condition failures. + html += renderTurnFailuresAsGrades(c.turn_results); } html += '
'; content.innerHTML = html; } + function renderTurnFailuresAsGrades(turns) { + var failed = turns.filter(function(tr) { return tr.status === "failed" || tr.status === "error"; }); + var passed = turns.filter(function(tr) { return tr.status === "completed"; }); + var total = failed.length + passed.length; + var rate = total === 0 ? 0 : Math.round((passed.length / total) * 100); + var badgeClass = rate === 100 ? "grade-pass" : "grade-fail"; + var html = '
' + rate + '%'; + html += '' + passed.length + ' passed, ' + failed.length + ' failed of ' + total + ' (post-condition checks)
'; + html += '
    '; + for (var i = 0; i < turns.length; i++) { + var tr = turns[i]; + if (tr.status === "failed" || tr.status === "error") { + html += '
  • \u2717'; + html += 'Turn ' + tr.turn_number + ' post-condition'; + if (tr.reason) html += '
    ' + esc(tr.reason) + '
    '; + html += '
  • '; + } else if (tr.status === "completed") { + html += '
  • \u2713'; + html += 'Turn ' + tr.turn_number + ' post-condition'; + html += '
  • '; + } + } + html += '
'; + return html; + } + function renderGradingBlock(g) { if (!g) return '
No grading data
'; var s = g.summary; diff --git a/internal/runner/runner.go b/internal/runner/runner.go index 76743d6b..af1e3474 100644 --- a/internal/runner/runner.go +++ b/internal/runner/runner.go @@ -446,9 +446,11 @@ func evalResultToCaseResult(res *evaluator.EvalResult) report.CaseResult { InputTokens: res.InputTokens, OutputTokens: res.OutputTokens, Grading: res.Grading, + JudgeSkills: res.JudgeSkills, Configuration: res.Configuration, Prompt: res.Prompt, Response: responseContent(res), + TurnResults: evalTurnResultsToReport(res.TurnResults), } if res.Error != nil { cr.Error = res.Error.Error() @@ -456,6 +458,23 @@ func evalResultToCaseResult(res *evaluator.EvalResult) report.CaseResult { return cr } +func evalTurnResultsToReport(turns []evaluator.TurnResult) []report.CaseTurnResult { + if len(turns) == 0 { + return nil + } + out := make([]report.CaseTurnResult, len(turns)) + for i, tr := range turns { + out[i] = report.CaseTurnResult{ + TurnNumber: tr.TurnNumber, + Content: tr.Content, + Response: tr.Response, + Status: string(tr.Status), + Reason: tr.Reason, + } + } + return out +} + func responseContent(res *evaluator.EvalResult) string { if res == nil || res.SessionResult == nil { return "" diff --git a/proposals/0002-agent-judge-specific-skill.md b/proposals/0002-agent-judge-specific-skill.md new file mode 100644 index 00000000..496c2190 --- /dev/null +++ b/proposals/0002-agent-judge-specific-skill.md @@ -0,0 +1,684 @@ +--- +title: agent_judge Judge-Specific Skill Support +authors: + - "kongtang" +creation-date: 2026-07-07 +last-updated: 2026-07-07 +status: draft +--- + +# SUP-0002: agent_judge Judge-Specific Skill Support + +Language: English | [中文](zh/0002-agent-judge-specific-skill.md) + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Requirements](#requirements) +- [Proposal](#proposal) + - [User Scenario Quick Reference](#user-scenario-quick-reference) + - [Notes, Constraints, and Caveats](#notes-constraints-and-caveats) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Configuration Schema](#configuration-schema) + - [Configuration Merge Semantics](#configuration-merge-semantics) + - [Path Resolution and Install Target](#path-resolution-and-install-target) + - [Agent Installation Adaptation and Progressive Loading](#agent-installation-adaptation-and-progressive-loading) + - [Mandatory Use Semantics](#mandatory-use-semantics) + - [Evaluator Execution Flow](#evaluator-execution-flow) + - [Isolation Semantics](#isolation-semantics) + - [Report Metadata](#report-metadata) + - [Documentation and Template Updates](#documentation-and-template-updates) +- [Test Plan](#test-plan) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed](#infrastructure-needed) +- [Upgrade & Migration Strategy](#upgrade--migration-strategy) + + +## Summary + +`agent_judge` can currently describe grading standards only through `judge.criteria`; it cannot install a Skill dedicated to the judge agent. For evaluations that need domain knowledge, reusable rubrics, strict output formats, or long-lived grading guidance, this forces authors to place large amounts of judging logic into every case YAML file. It also bypasses each Agent Engine's native Skill progressive-loading mechanism, increasing the risk of filling the context window with large rubric documents. + +This proposal, based on [GitHub issue #134](https://github.com/alibaba/skill-up/issues/134), adds a `judge.skills` configuration so `agent_judge` can install one or more judge-specific Skills through the selected Agent adapter's native Skill installation path. The feature keeps judge Skills isolated from the Skill under test, preserves existing `judge.criteria`, benchmark `with_skill` / `without_skill`, and Agent execution semantics, and records the configured judge Skill information in reports. + +## Motivation + +`agent_judge` is useful when semantic understanding is required, but today's configuration surface is limited to natural-language criteria, model selection, pass threshold, and timeout. Eval authors commonly run into these problems: + +1. **Rubrics are too long for YAML**: domain rules, style guides, scoring details, and negative examples are awkward to maintain inline. +2. **Judging logic needs reuse**: when many evals or cases share the same rubric, duplicated criteria drift over time. +3. **The judge agent needs different context**: the run agent should install the Skill under test, while the judge agent may need a separate Skill that teaches it how to grade. +4. **Judge prompts should evolve independently**: a judge Skill can version and refine its rubric without editing every case file. +5. **Isolation matters**: judge helper Skills must not leak into the run agent, especially in benchmark mode where they would pollute the measured capability. +6. **Progressive loading is the point**: the goal is not to concatenate `SKILL.md`, `references/`, and `assets/` into the judge prompt. The goal is to rely on the Agent's own Skill discovery, selection, and on-demand loading behavior so long rubrics enter context only when needed. +7. **Reports need auditability**: results should say which judge Skills were configured, otherwise reviewers cannot understand the grading basis or reproduce the judge environment. + +The current code path exposes the gap: + +- `config.JudgeConfig` contains fields such as `type`, `model`, `criteria`, `pass_threshold`, and `timeout_seconds`, but no judge-level Skill reference. +- `defaultEvaluator.setupCaseEnvironment()` installs `evalCfg.Skills` only for the main run agent. +- `agent_judge` later creates or resolves a judge agent in `resolveJudgeAgent()`, wraps it through `judge.NewJudge()`, and `AgentJudge.Evaluate()` calls `judgeAgent.Run()`. +- There is no judge-phase step that calls `judgeAgent.InstallSkill(...)`. + +### Goals + +1. **Support judge-level Skill configuration**: add a `skills` field to `JudgeConfig`, reusing existing `SkillRef` semantics. +2. **Apply only to `agent_judge`**: `judge.skills` is meaningful only when `judge.type: agent_judge`. +3. **Install into the judge agent**: install `judge.skills` into the judge agent runtime before `AgentJudge.Evaluate()` calls `Run()`. +4. **Keep run and judge agents isolated**: `judge.skills` is not installed into the main run agent, and `eval.skills` is not automatically installed into the judge agent. +5. **Preserve benchmark semantics**: `judge.skills` is installed for both `with_skill` and `without_skill` because it is grading tooling, not the Skill under test. +6. **Remain backward compatible**: existing `agent_judge` configurations behave the same when `judge.skills` is absent. +7. **Preserve native Agent Skill mechanisms**: different Agent Engines may have different Skill directories, manifests, indexes, and discovery mechanisms. skill-up should trigger installation through the Agent adapter's `InstallSkill` abstraction, not inject Skill documents into prompts. +8. **Require judge Skill usage**: when users configure `judge.skills` for `agent_judge`, the judge prompt must explicitly instruct the judge agent to use the installed Skills as authoritative grading guidance. Installation is necessary but not sufficient. +9. **Make reports auditable**: report the judge Skill metadata used for each judged result. +10. **Cover tests and docs**: add coverage for config loading, validation, merging, installation isolation, mandatory usage prompt behavior, report metadata, and documentation examples. + +### Non-Goals + +1. **No new Skill package manager**: this proposal reuses `SkillRef` / `runtime.SkillConfig` and does not design registry download, version locking, or dependency resolution. +2. **No change to main run Skill installation**: `eval.skills` continues to mean Skills needed by the run agent or the Skill under test. +3. **No rewrite of the `AgentJudge` scoring protocol**: this phase continues to use criteria-driven JSON result parsing. +4. **Judge Skills do not define the criteria list by themselves**: Skills may provide detailed rubrics and constraints, but structured scoring dimensions still come from `judge.criteria`. +5. **No Skill installation for non-`agent_judge` judges**: `rule_based` and `script` judges do not read `judge.skills`. +6. **No prompt-concatenation fallback**: if an Agent adapter cannot install Skills, this proposal does not allow reading Skill files and appending their contents to the judge prompt as a substitute. + +## Requirements + +### Must Have + +| ID | Requirement | Acceptance Criteria | +| --- | --- | --- | +| R1 | `JudgeConfig` supports `skills` | YAML can declare a `skills` array under `judge:` and load it into config | +| R2 | Only `agent_judge` can use it | `rule_based` / `script` with `judge.skills` fails validation with a clear error | +| R3 | Install before judging | Configured judge Skills are installed before `AgentJudge.Evaluate()` invokes the judge agent | +| R4 | Installation isolation | The run agent receives only `eval.skills`; the judge agent receives `judge.skills` | +| R5 | Benchmark does not suppress judge Skills | `without_skill` skips `eval.skills` but still installs `judge.skills` | +| R6 | Consistent path resolution | Local judge Skill paths resolve relative to the Skill root, consistent with `eval.skills` | +| R7 | Backward compatibility | Existing configs require no changes and existing `judge.criteria` behavior remains | +| R8 | Native Skill progressive loading | Implementation calls the judge Agent adapter's `InstallSkill`; it must not read Skill docs and concatenate them into the judge prompt | +| R9 | Report judge Skills | JSON/HTML reports show the judge Skill list; JUnit exposes it at least through properties | +| R10 | Mandatory judge Skill use | When `judge.skills` is non-empty, the prompt sent by `AgentJudge` must include a mandatory-use instruction and Skill identifiers | + +### Should Have + +| ID | Requirement | Acceptance Criteria | +| --- | --- | --- | +| S1 | Multiple judge Skills | `judge.skills` can declare multiple Skills, installed in configuration order | +| S2 | Case-level override | Case-level `judge.skills` can override eval-level judge configuration | +| S3 | Clear diagnostics | Installation failures include the judge Skill path and judge-phase context | +| S4 | Documentation updates | English/Chinese writing-evals docs and skill-upper references include examples | +| S5 | Document Agent differences | Docs explain that judge Skill installation depends on each adapter's Skill support and does not guarantee identical behavior across Agents | + +### Nice to Have + +| ID | Requirement | Acceptance Criteria | +| --- | --- | --- | +| N1 | Richer report metadata | Reports may include judge Skill digest, target, and install status without exposing sensitive absolute paths | +| N2 | Future skill-only judge compatibility | Future work can allow a judge Skill to provide default criteria without breaking this design | + +## Proposal + +Add `judge.skills`, using the same plural form as top-level `skills`: + +```yaml +judge: + type: agent_judge + model: anthropic/claude-sonnet-4-6 + skills: + - source: local_path + path: evals/fixtures/judge-skill + criteria: + - "The answer is correct according to the rubric in the installed judge Skill" +``` + +High-level flow: + +1. Parse `judge.skills` from `eval.yaml` and case YAML. +2. Validate that `judge.skills` is used only with `agent_judge`. +3. Execute the case as usual: prepare runtime, install the run agent, install MCP, and install `eval.skills` for the run agent. +4. Run the main agent and collect `SessionResult`, workspace diff, transcript, and other judge inputs. +5. Enter the judge phase and resolve or create the judge agent. +6. Install merged `judge.skills` into the judge agent through that Agent adapter's `InstallSkill`. +7. When building the judge prompt, if `judge.skills` is non-empty, add an instruction that requires the judge agent to use the installed judge Skills. Do not grade as ordinary criteria-only `agent_judge`. +8. Run the judge agent and parse the structured JSON grading result. +9. Write judge Skill metadata into reports. + +``` +Case Runtime + +setupCaseEnvironment + - install run agent + - install MCP + - install eval.skills --------------+ + | + Main Run Agent + runs Skill under test + | + transcript/diff/output + | +judge phase | + - resolve judge agent | + - install judge.skills -----+ | + - AgentJudge.Evaluate | | + v v + Judge Agent + grades with installed + judge Skill + criteria +``` + +### User Scenario Quick Reference + +#### Scenario 1: Reusable Domain Rubric + +```yaml +schema_version: v1alpha1 + +skills: + - source: local_path + path: . + +judge: + type: agent_judge + model: anthropic/claude-sonnet-4-6 + skills: + - source: local_path + path: evals/fixtures/sql-judge-skill + criteria: + - "The SQL change satisfies the safety and compatibility rules defined by the judge Skill" + - "The grading decision cites concrete evidence rather than generic opinions" +``` + +The run agent installs the Skill under test. The judge agent installs `sql-judge-skill`, which contains database review rules, counterexamples, and output requirements. + +#### Scenario 2: Case-Level Judge Skill Override + +```yaml +# evals/eval.yaml +judge: + type: agent_judge + model: anthropic/claude-sonnet-4-6 + skills: + - source: local_path + path: evals/fixtures/default-judge-skill + criteria: + - "Grade output quality according to the default judge Skill" +``` + +```yaml +# evals/cases/security-review.yaml +judge: + type: agent_judge + skills: + - source: local_path + path: evals/fixtures/security-judge-skill + criteria: + - "Grade whether the answer identifies high-risk issues according to the security judge Skill" +``` + +When a case declares its own `judge.type`, the case-level judge config is treated as a complete judge strategy: `skills` and `criteria` come from the case, while `model`, `pass_threshold`, and `timeout_seconds` can still inherit from global defaults. + +#### Scenario 3: Stable Grading Tooling in Benchmark Mode + +```yaml +benchmark: + enabled: true + +skills: + - source: local_path + path: . + +judge: + type: agent_judge + model: anthropic/claude-sonnet-4-6 + skills: + - source: local_path + path: evals/fixtures/judge-rubric + criteria: + - "Grade whether the output satisfies the acceptance criteria according to judge-rubric" +``` + +Benchmark execution: + +- `with_skill`: the run agent installs `eval.skills`, and the judge agent installs `judge.skills`. +- `without_skill`: the run agent skips `eval.skills`, and the judge agent still installs `judge.skills`. + +This compares the effect of the Skill under test, not whether the grading tool exists. + +### Notes, Constraints, and Caveats + +1. **`criteria` remains required**: in this phase, `judge.criteria` still defines structured scoring dimensions. A judge Skill may contain long rubrics, but YAML keeps at least one criterion so result count and report structure remain deterministic. +2. **Reuse `SkillRef`**: `judge.skills` uses `source`, `path`, and `target`; there is no parallel singular `judge.skill` syntax. +3. **Must rely on Agent Skill mechanisms**: judge Skills are valuable because Agents can discover and load them on demand. Implementation must not read an entire Skill directory into `criteria` or the judge prompt. +4. **Agent installation methods may differ**: Claude Code, Codex, Qoder CLI, and custom Agents may use different Skill locations or indexes. skill-up hands the same `runtime.SkillConfig` to the relevant adapter. +5. **Local paths first**: phase one uses the existing local Skill installation capability. If top-level `skills` later supports registries, `judge.skills` can reuse that path. +6. **Installation failure is ERROR**: failed judge Skill installation means the judge environment is not ready; mark the case as ERROR, not FAIL. +7. **No silent fallback**: if `judge.skills` is configured but cannot be installed, do not continue with an unskilled judge agent. + +### Risks and Mitigations + +| Risk | Impact | Probability | Mitigation | +| --- | --- | --- | --- | +| Judge Skill accidentally installs into the run agent | Benchmark results are polluted | Medium | Installation code receives only `judgeAgent`; tests record run/judge installs separately | +| `without_skill` skips judge Skills | Baseline cannot be graded by the same rubric | Medium | Judge Skill installation does not depend on `configName` | +| Case/global merge semantics are unclear | Authors cannot predict which judge Skill is used | Medium | Reuse existing full override semantics for case-level `judge.type` and document them | +| `criteria` conflicts with judge Skill rubric | Grading becomes unstable | Medium | Docs recommend stable criteria dimensions and detailed rubrics in Skills | +| Skill path escapes the Skill root | Unexpected local files may be read | Low | Reuse or strengthen path validation so resolved relative paths remain under the Skill root | +| Prompt-concatenation fallback for compatibility | Loses progressive loading and can exhaust context | Medium | Explicitly forbid prompt-concatenation fallback; unsupported adapters return ERROR | +| Skill discovery differs by Agent | Same judge Skill may behave differently across engines | Medium | Keep adapter-level tests and record engine plus judge Skill metadata in reports | +| Reports omit judge Skill info | Grading basis is not auditable | Medium | Add judge Skill metadata to EvalResult/report generation | +| Judge Skill is installed but not used | Grading still follows ordinary criteria, ignoring the user-defined rubric | Medium | `AgentJudge` prompt must require use of installed judge Skills; unit tests assert the prompt and fixtures verify behavior | + +## Design Details + +### Configuration Schema + +Extend `JudgeConfig` in `internal/config/schema.go`: + +```go +// JudgeConfig describes the evaluation strategy. +type JudgeConfig struct { + Type string `json:"type" yaml:"type"` + ScriptPath string `json:"script_path,omitempty" yaml:"script_path,omitempty"` + Model string `json:"model,omitempty" yaml:"model,omitempty"` + Criteria []string `json:"criteria,omitempty" yaml:"criteria,omitempty"` + Skills []SkillRef `json:"skills,omitempty" yaml:"skills,omitempty"` + + PassThreshold *float64 `json:"pass_threshold,omitempty" yaml:"pass_threshold,omitempty"` + TimeoutSeconds *int `json:"timeout_seconds,omitempty" yaml:"timeout_seconds,omitempty"` + Success []Rule `json:"success,omitempty" yaml:"success,omitempty"` + Failure []Rule `json:"failure,omitempty" yaml:"failure,omitempty"` +} +``` + +Validation rules: + +1. If `judge.skills` is non-empty, `judge.type` must be `agent_judge`. +2. Each `judge.skills[*].source` should currently be `local_path` or another source already supported by top-level `skills`. +3. For `source: local_path`, `path` is required and must not be blank. +4. `target` is optional and follows top-level `skills[*].target` semantics. +5. `agent_judge` still requires `model` and at least one `criteria` entry unless a later proposal changes the `AgentJudge` protocol. +6. Checks that depend on inherited judge defaults, especially the required `agent_judge` `model`, must run against the effective judge config after `judge.MergeJudgeConfig(global, caseLevel)`. Raw case validation may continue to check case-local constraints, but it must not reject a case-level `agent_judge` only because `model` is inherited from the global judge config. + +### Configuration Merge Semantics + +Reuse the current `judge.MergeJudgeConfig(global, caseLevel)` behavior: + +- If a case does not declare `judge.type`, use the global judge config, including global `judge.skills`. +- If a case declares `judge.type`, treat the case judge config as a full override. `skills`, `criteria`, `success`, `failure`, and `script_path` come from the case. +- `model`, `pass_threshold`, and `timeout_seconds` may continue inheriting from global config because current logic already does this. + +No implicit append behavior is added for the new field. A judge Skill usually represents a complete grading context; silently merging global and case-level Skills can create rubric conflicts. If an author needs multiple Skills, the case-level `judge.skills` should explicitly list all of them. + +### Path Resolution and Install Target + +Local judge Skill path resolution matches top-level `skills`: + +```go +skillSourceDir := e.loader.SkillDir() +skillSource := filepath.Join(skillSourceDir, judgeSkillRef.Path) +skillCfg := runtime.SkillConfig{ + Source: skillSource, + Target: judgeSkillRef.Target, +} +``` + +Implementation should extract a small shared helper, for example: + +```go +func resolveSkillConfig(skillDir string, ref config.SkillRef) runtime.SkillConfig { + return runtime.SkillConfig{ + Source: filepath.Join(skillDir, ref.Path), + Target: ref.Target, + } +} +``` + +Notes: + +- Relative paths are resolved from the Skill root, not the current working directory or `eval.yaml` directory. +- Installation order follows the `judge.skills` array order. +- Error messages should include `judge.skills[i].path`. + +### Agent Installation Adaptation and Progressive Loading + +Different Agents may have different Skill conventions: + +- Claude Code may have its own Skill directory and indexing rules. +- Codex may have its own Skill search, resource discovery, and loading conventions. +- Qoder CLI may use a different install command, target directory, or manifest handling. +- Custom Agents may implement installation through `InstallSkillCmd` or their own protocol. + +The evaluator should not understand each Agent's internal Skill layout. It does only three things: + +1. Resolve `runtime.SkillConfig{Source, Target}` from config. +2. Call `InstallSkill(ctx, rt, skillCfg)` on the final judge agent at the correct phase. +3. Record judge Skill metadata for reports and diagnostics. + +The adapter owns the concrete installation method: + +```go +type Agent interface { + Run(ctx context.Context, rt Runtime, opts ExecOptions, messages []Message) (*SessionResult, error) + Install(ctx context.Context, rt Runtime) error + InstallMCP(ctx context.Context, rt Runtime, cfg runtime.MCPConfig) error + InstallSkill(ctx context.Context, rt Runtime, skillCfg runtime.SkillConfig) error + CheckCredentials(ctx context.Context) error + Name() string +} +``` + +Constraints: + +- `InstallSkill` should use the Agent's native Skill installation path: directory layout, index files, manifest handling, cache refresh, or CLI install entrypoint. +- The evaluator must not read judge Skill `SKILL.md`, `references/`, or `assets/`, and must not insert these file contents into the judge prompt. +- If an adapter cannot install Skills, it returns a clear error. When `judge.skills` is configured, do not silently fall back to appending Skill docs to the prompt. +- Custom Agents continue to use the existing `InstallSkillCmd` capability. If no installation capability exists but `judge.skills` is configured, the judge phase fails with ERROR. + +This preserves the core value of Skills: the Agent can progressively load relevant instructions and resources during judging instead of consuming the whole context window up front. + +### Mandatory Use Semantics + +Installing a judge Skill does not guarantee `agent_judge` will use it. To ensure user-defined judge Skills actually participate in grading, `AgentJudge` prompt construction must add a mandatory-use block when `judge.skills` is non-empty, for example: + +```text +You MUST use the installed judge Skill(s) listed below as the authoritative +grading rubric before evaluating the case. Do not grade this case using only +the inline criteria. The inline criteria identify the result dimensions, while +the judge Skill(s) define the detailed rubric, constraints, and evidence rules. + +Installed judge Skill(s): +- evals/fixtures/judge-skill +``` + +This block references only Skill identifiers and configured paths; it does not expand Skill file contents. Its purpose is to trigger the Agent's Skill selection mechanism and make grading priority explicit: + +1. `judge.skills` provides detailed rubrics, constraints, style guides, and evidence requirements. +2. `judge.criteria` provides structured result dimensions and determines the result count. +3. If they conflict, the prompt should state that the judge Skill rubric is authoritative unless a criterion defines a more specific case-level acceptance condition. + +Implementation requirements: + +- `AgentJudge.buildPrompt()` or an equivalent function receives resolved judge Skill metadata. +- When `judge.skills` is non-empty, the prompt must include a mandatory-use instruction and a stable identifier for every Skill. +- When `judge.skills` is empty, prompt behavior remains criteria-only for backward compatibility. +- The evaluator does not have to prove which internal file the Agent loaded. It must make the observable input require Skill use, and integration fixtures should prove that a Skill-capable Agent uses the Skill under that instruction. +- Reports should distinguish "configured/installed judge skills" from any future "agent-acknowledged used judge skills". This proposal requires at least the former. + +### Evaluator Execution Flow + +Add judge Skill installation near `defaultEvaluator.newJudgeForCase()`: + +```go +func (e *defaultEvaluator) newJudgeForCase( + ctx context.Context, + rt runtime.Runtime, + judgeCfg config.JudgeConfig, + runAgent agent.Agent, +) (judge.Judge, error) { + judgeCfg = resolveJudgeScriptPath(e.judgeScriptBaseDir(), judgeCfg) + + judgeAgent, err := e.resolveJudgeAgent(ctx, judgeCfg, runAgent) + if err != nil { + return nil, err + } + + if err := e.installJudgeSkills(ctx, rt, judgeCfg, judgeAgent); err != nil { + return nil, err + } + + j, err := judge.NewJudge(judgeCfg, judgeAgent, rt) + if err != nil { + return nil, fmt.Errorf("failed to create judge: %w", err) + } + return j, nil +} +``` + +Core `installJudgeSkills` behavior: + +```go +func (e *defaultEvaluator) installJudgeSkills( + ctx context.Context, + rt runtime.Runtime, + judgeCfg config.JudgeConfig, + judgeAgent agent.Agent, +) error { + if judgeCfg.Type != "agent_judge" || len(judgeCfg.Skills) == 0 { + return nil + } + if e.loader == nil { + return errors.New("judge.skills requires a loader to resolve local paths") + } + + skillDir := e.loader.SkillDir() + for i, ref := range judgeCfg.Skills { + skillCfg := resolveSkillConfig(skillDir, ref) + if err := judgeAgent.InstallSkill(ctx, rt, skillCfg); err != nil { + return fmt.Errorf("failed to install judge skill judge.skills[%d].path=%q: %w", i, ref.Path, err) + } + logging.DebugContextf(ctx, "Evaluator: judge skill installed: %s", filepath.Base(skillCfg.Source)) + } + return nil +} +``` + +The installation point is after `resolveJudgeAgent()` and before `judge.NewJudge()` because: + +1. The final judge agent is known, so the Skill is not installed into the run agent by mistake. +2. The runtime exists and the main run output/workspace diff has already been prepared. +3. Installation failures can return a clear ERROR before `AgentJudge.Evaluate()` starts. + +### Isolation Semantics + +| Configuration | Install into run agent | Install into judge agent | +| --- | --- | --- | +| `eval.skills` + `with_skill` | yes | no | +| `eval.skills` + `without_skill` | no | no | +| `judge.skills` + `with_skill` | no | yes | +| `judge.skills` + `without_skill` | no | yes | + +If the current implementation ever reuses the `runAgent` instance as the judge agent, installation should still follow logical role boundaries: `eval.skills` installs before the main run, and `judge.skills` installs before judging. For built-in engines, prefer a distinct judge agent for `agent_judge`. If reuse is unavoidable, docs and tests must ensure judge Skill installation cannot affect the already-completed main run phase. + +### Report Metadata + +Reports should record the judge Skills actually configured for the judging run. Add a non-sensitive structure to case result or grading metadata: + +```go +// JudgeSkillInfo describes a judge Skill used during agent_judge evaluation. +type JudgeSkillInfo struct { + Source string `json:"source,omitempty"` // e.g. local_path + Path string `json:"path,omitempty"` // config path, relative when configured that way + Target string `json:"target,omitempty"` + Name string `json:"name,omitempty"` // derived from path basename or Skill metadata +} +``` + +Report expectations: + +- JSON report: include `judge_skills` or an equivalent field under each case/config result. +- HTML report: show judge Skill name, configured path, and target in case details. +- JUnit report: expose basic fields through testcase properties such as `judge.skills.count` and `judge.skills..path`. +- Anthropic `grading.json`: if no compatible extension point exists, it can omit these official grading fields, but skill-up's JSON/HTML reports must retain the metadata. + +Security and privacy: + +- Record configured relative `path` by default, not expanded local absolute paths. +- Do not record Skill file contents. +- Future digests may help reproduction, but large file contents should not be embedded in reports. + +### Documentation and Template Updates + +Update: + +- `docs/guide/writing-evals.md`: add a `skills` example to the `judge: agent_judge` section and explain isolation. +- `docs/zh/guide/writing-evals.md`: add the Chinese equivalent. +- `skills/skill-upper/references/eval-yaml.md`: document `judge.skills`. +- `skills/skill-upper/references/judge-types.md`: explain when to use judge Skills versus inline criteria. +- Agent or custom-engine docs: explain that judge Skill installation depends on adapter `InstallSkill` capability and will not degrade to prompt concatenation. +- Templates with `agent_judge` examples may include commented examples, but should not enable judge Skills by default because they increase cost and complexity. + +## Test Plan + +### Unit Tests + +1. **Config loading** + - Eval-level `judge.skills` deserializes into `JudgeConfig.Skills`. + - Case-level `judge.skills` deserializes. + - Multiple Skills preserve configuration order. + +2. **Config validation** + - `judge.type: agent_judge` + `judge.skills` passes. + - `judge.type: rule_based` + `judge.skills` fails. + - `judge.type: script` + `judge.skills` fails. + - Blank `judge.skills[*].path` fails. + - Missing `criteria` for `agent_judge` keeps the existing failure behavior. + +3. **Config merge** + - A case without `judge.type` inherits global `judge.skills`. + - A case with its own `judge.type` and `judge.skills` does not append global `judge.skills`. + - A case with `judge.type` but no `model` still inherits global `model`. + +4. **Evaluator installation behavior** + - Mock run agent receives only `eval.skills`. + - Mock judge agent receives only `judge.skills`. + - `without_skill` skips `eval.skills` but installs `judge.skills`. + - Judge Skill install failure marks the case ERROR and does not invoke judge agent `Run()`. + - Unsupported adapter + configured `judge.skills` returns ERROR and does not concatenate Skill docs into the prompt. + +5. **Mandatory-use prompt** + - Non-empty `judge.skills` makes the `AgentJudge` prompt include a mandatory-use instruction. + - The prompt includes a stable identifier or configured path for each judge Skill. + - The prompt does not include judge Skill file contents. + - Empty `judge.skills` preserves existing criteria-only behavior. + +6. **Path resolution** + - Relative paths resolve from the Skill root. + - `target` passes through to `runtime.SkillConfig.Target`. + +7. **Report metadata** + - JSON report contains the judge Skill list. + - HTML report displays judge Skill name and configured path. + - JUnit properties include judge Skill count and path. + - Reports do not contain Skill file contents or local absolute paths. + +### Integration / E2E-Style Fixture + +Add a fixture such as: + +```text +e2e/testdata/agent-judge-skill/ + SKILL.md + evals/ + eval.yaml + cases/ + uses-judge-skill.yaml + fixtures/ + judge-skill/ + SKILL.md +``` + +Use a controlled mock/custom agent to verify: + +- The run agent does not read the judge Skill. +- The judge prompt explicitly requires use of the installed judge Skill. +- The judge agent reads the judge Skill through its Skill mechanism, not from prompt body text. +- Missing judge Skill fails or errors the same case, proving installation matters. +- The mock/custom agent relies on its own Skill discovery mechanism; tests do not pass because `SKILL.md` was appended to the prompt. +- Generated reports contain judge Skill metadata. + +## Drawbacks + +1. **Configuration surface grows**: `judge.skills` needs clear docs explaining how it differs from top-level `skills`. +2. **Reviewers must inspect Skill files**: rubrics move from YAML into Skill files, so review requires looking at both. +3. **Additional install cost**: each `agent_judge` may perform one or more extra Skill installs. +4. **`criteria` still remains**: this phase does not fully implement "Skill defines all scoring dimensions", so users may still need a short criterion. +5. **Cross-Agent behavior may differ**: Skill discovery and loading are inherently Agent-specific. +6. **Reports need extension work**: JSON/HTML/JUnit report paths need to carry judge Skill metadata. + +## Alternatives + +### Alternative A: Singular `judge.skill` + +```yaml +judge: + type: agent_judge + skill: + source: local_path + path: evals/fixtures/judge-skill +``` + +This is concise for one Skill, but top-level config already uses `skills`. Adding a singular form creates parallel semantics and a future migration when multiple judge Skills are needed. Not chosen. + +### Alternative B: Put Judge Skill in Top-Level `skills` + +```yaml +skills: + - source: local_path + path: . + - source: local_path + path: evals/fixtures/judge-skill +``` + +This avoids schema changes, but installs the judge Skill into the run agent, polluting the measured capability and breaking benchmark interpretation. Not chosen. + +### Alternative C: External Markdown Criteria File + +```yaml +judge: + type: agent_judge + criteria_file: evals/fixtures/rubric.md +``` + +This is simpler, but it cannot use Agent Skill conventions, directory structure, resources, installation, or progressive loading. If implemented by reading Markdown and appending it to the prompt, it places the long rubric into context all at once, which contradicts the core motivation for using Skills. Not chosen. + +### Alternative C2: Prompt-Concatenation Fallback on Install Failure + +This appears compatible, but degrades Skill behavior into an ordinary long prompt: + +- It loses on-demand loading. +- It can fill context with `references/` and asset descriptions. +- It cannot validate that the judge Skill works through the real Agent Skill mechanism. +- Reports cannot cleanly distinguish "installed Skill" from "concatenated documents". + +This fallback is explicitly not chosen. + +### Alternative D: Let Judge Skill Replace `criteria` + +```yaml +judge: + type: agent_judge + skills: + - source: local_path + path: evals/fixtures/judge-skill +``` + +This is a useful future direction, but it requires changing the `AgentJudge` prompt and JSON result parser because current report structure validates the number of results against `criteria`. This proposal keeps `criteria` required to reduce implementation risk. + +## Infrastructure Needed + +No external service or third-party dependency is required. Implementation reuses existing config loading, runtime Skill installation, Agent interfaces, and test infrastructure. + +Useful local test assets: + +- A judge Skill fixture. +- A mock judge agent that records `InstallSkill` calls. +- A custom/mock agent fixture for integration-style verification. + +## Upgrade & Migration Strategy + +This change is backward compatible: + +- Existing `agent_judge` configs require no changes. +- Behavior is unchanged when `judge.skills` is absent. +- Top-level `skills` semantics remain unchanged. +- Benchmark output semantics remain unchanged; the judge side can now have stable rubric tooling. + +Recommended migration: + +1. Move long judge rubrics into `evals/fixtures/-judge-skill/SKILL.md`. +2. Reference that path from `judge.skills`. +3. Reduce `judge.criteria` to stable scoring dimensions, for example "judge according to the installed judge Skill's safety rules". +4. Run `skill-up validate` and a small eval to confirm the judge agent can discover and use the judge Skill. diff --git a/proposals/README.md b/proposals/README.md index 99756ce6..076d49e6 100644 --- a/proposals/README.md +++ b/proposals/README.md @@ -7,3 +7,4 @@ This is the complete list of skill-up Enhancement Proposals: | Proposal | Title | Chinese Version | Status | Last Updated | | :----------------------------------------------: | :--------------------------------------------: | :-------------: | :---------: | :----------: | | [SUP-0001](0001-multi-turn-conversation-eval.md) | Multi-Turn Conversation Evaluation Support | [中文](zh/0001-multi-turn-conversation-eval.md) | provisional | 2026-05-19 | +| [SUP-0002](0002-agent-judge-specific-skill.md) | agent_judge Judge-Specific Skill Support | [中文](zh/0002-agent-judge-specific-skill.md) | draft | 2026-07-07 | diff --git a/proposals/zh/0002-agent-judge-specific-skill.md b/proposals/zh/0002-agent-judge-specific-skill.md new file mode 100644 index 00000000..beabf376 --- /dev/null +++ b/proposals/zh/0002-agent-judge-specific-skill.md @@ -0,0 +1,693 @@ +--- +title: agent_judge 专用评审 Skill 支持 +authors: + - "kongtang" +creation-date: 2026-07-07 +last-updated: 2026-07-07 +status: draft +--- + +# SUP-0002: agent_judge 专用评审 Skill 支持 + +语言:[English](../0002-agent-judge-specific-skill.md) | 中文 + + +- [摘要](#摘要) +- [动机](#动机) + - [目标](#目标) + - [非目标](#非目标) +- [需求](#需求) +- [提案](#提案) + - [用户场景速查](#用户场景速查) + - [注意事项、约束与说明](#注意事项约束与说明) + - [风险与缓解措施](#风险与缓解措施) +- [设计细节](#设计细节) + - [配置 Schema](#配置-schema) + - [配置合并语义](#配置合并语义) + - [路径解析与安装目标](#路径解析与安装目标) + - [Agent 安装适配与渐进式加载](#agent-安装适配与渐进式加载) + - [强制使用语义](#强制使用语义) + - [评估器执行流程](#评估器执行流程) + - [隔离语义](#隔离语义) + - [报告元数据](#报告元数据) + - [文档与模板更新](#文档与模板更新) +- [测试计划](#测试计划) +- [缺点](#缺点) +- [替代方案](#替代方案) +- [所需基础设施](#所需基础设施) +- [升级与迁移策略](#升级与迁移策略) + + +## 摘要 + +`agent_judge` 目前只能通过 `judge.criteria` 描述评审标准,无法为评审 Agent 安装专门的 Skill。对于需要领域知识、可复用评分规约、严格输出格式或长期维护 rubric 的评估,这会迫使作者把大量评审逻辑塞进每个 case 的 YAML 中,也会绕过各 Agent 的 Skill 渐进式加载机制,增加 context 被大文档撑爆的风险。 + +本提案根据 [GitHub issue #134](https://github.com/alibaba/skill-up/issues/134) 设计 `judge.skills` 配置,使 `agent_judge` 可以在评审阶段通过对应 Agent 的原生 Skill 安装方式安装一个或多个专用评审 Skill。该能力与被测 Skill 的安装路径隔离,保持现有 `judge.criteria`、benchmark `with_skill` / `without_skill` 和 Agent 运行流程向后兼容,并在报告中记录实际使用的 judge Skill 信息。 + +## 动机 + +`agent_judge` 适合处理无法用确定性规则表达的语义评估,但当前配置只有自然语言 criteria、模型、阈值和超时。实际 eval 作者常遇到以下问题: + +1. **评审规则过长**:领域规范、风格指南、评分细则和反例库不适合长期内嵌在 YAML 中。 +2. **评审逻辑需要复用**:多个 eval 或多个 case 共享同一套 rubric 时,重复 criteria 容易漂移。 +3. **评审 Agent 需要不同上下文**:主运行 Agent 应安装被测 Skill,而 judge Agent 可能需要一个完全不同的评审 Skill。 +4. **评审提示需要独立演进**:评审 Skill 可以随版本迭代,不必修改所有 case 文件。 +5. **隔离性要求明确**:评审辅助 Skill 不能泄漏给主运行 Agent,否则 benchmark 会把评分工具错误地变成被测能力的一部分。 +6. **需要渐进式加载**:使用 Skill 的目的不是把 `SKILL.md`、references、assets 等内容全部拼进 judge prompt,而是依赖 Agent 自身的 Skill 发现、选择和按需加载能力,让长 rubric 在需要时才进入上下文。 +7. **需要可审计性**:评审结果应说明用了哪些 judge Skill,否则报告读者难以判断评分依据和复现环境。 + +当前代码路径也体现了这个缺口: + +- `config.JudgeConfig` 已包含 `type`、`model`、`criteria`、`pass_threshold`、`timeout_seconds` 等字段,但没有 judge 级 Skill 引用。 +- `defaultEvaluator.setupCaseEnvironment()` 只在主运行 Agent 上安装 `evalCfg.Skills`。 +- `agent_judge` 在评审阶段通过 `resolveJudgeAgent()` 创建或复用 judge Agent,然后 `judge.NewJudge()` 包装为 `AgentJudge`,最终由 `AgentJudge.Evaluate()` 调用 `judgeAgent.Run()`。 +- 评审阶段没有调用 `judgeAgent.InstallSkill(...)` 的安装步骤。 + +### 目标 + +1. **支持 judge 级 Skill 配置**:在 `JudgeConfig` 中新增 `skills` 字段,复用现有 `SkillRef` 语义。 +2. **仅作用于 `agent_judge`**:`judge.skills` 只在 `judge.type: agent_judge` 时生效。 +3. **安装到 judge Agent**:在 `AgentJudge.Evaluate()` 调用 `Run()` 前,把 `judge.skills` 安装到 judge Agent 所在 runtime。 +4. **与主运行 Agent 隔离**:`judge.skills` 不安装到主运行 Agent;`eval.skills` 也不自动安装到 judge Agent。 +5. **保持 benchmark 语义**:无论当前配置变体是 `with_skill` 还是 `without_skill`,judge Skill 都应安装,因为它是评分工具,不是被测 Skill。 +6. **保持向后兼容**:不改变已有 `agent_judge` 配置的行为;没有 `judge.skills` 的 eval 无需迁移。 +7. **保留 Agent 原生 Skill 机制**:不同 Agent 的 Skill 安装目录、manifest、索引或发现机制可能不同,skill-up 只通过 Agent adapter 的 `InstallSkill` 抽象触发安装,不把 Skill 文档展开注入 prompt。 +8. **强制使用 judge Skill**:当用户为 `agent_judge` 配置 `judge.skills` 时,评审 prompt 必须显式要求 judge Agent 使用这些已安装 Skill 作为权威评审依据;安装不是充分条件。 +9. **报告可审计**:报告中记录本次评审配置使用的 judge Skill 元数据。 +10. **覆盖测试与文档**:补充配置加载、校验、合并、安装隔离、强制使用、报告元数据和文档示例。 + +### 非目标 + +1. **不引入新的 Skill 包管理系统**:本提案复用现有 `SkillRef` / `runtime.SkillConfig`,不设计 registry 下载、版本锁定或依赖解析。 +2. **不改变主运行 Agent 的 Skill 安装语义**:`eval.skills` 仍然只表示被测 Skill 或主运行 Agent 所需 Skill。 +3. **不重写 `AgentJudge` 评分协议**:本阶段继续使用 criteria 驱动的 JSON 评分结果解析。 +4. **不让 judge Skill 自动决定 criteria 列表**:Skill 可以提供 rubric 和输出约束,但结构化评分项仍由 `judge.criteria` 声明。 +5. **不支持非 `agent_judge` 的 Skill 安装**:`rule_based` 和 `script` judge 不读取 `judge.skills`。 +6. **不提供 prompt 拼接降级方案**:如果某个 Agent adapter 不支持 Skill 安装,本提案不允许把 Skill 文件内容直接拼进 judge prompt 作为替代实现。 + +## 需求 + +### 必须有 + +| ID | 需求 | 验收标准 | +| --- | --- | --- | +| R1 | `JudgeConfig` 支持 `skills` | YAML 中可在 `judge:` 下声明 `skills` 数组,并被加载到配置结构 | +| R2 | 仅 `agent_judge` 可使用 | `rule_based` / `script` 配置 `judge.skills` 时校验失败或给出明确错误 | +| R3 | 评审前安装 | `AgentJudge.Evaluate()` 调用 judge Agent 前,配置的 judge Skills 已安装完成 | +| R4 | 安装隔离 | 主运行 Agent 只安装 `eval.skills`;judge Agent 只额外安装 `judge.skills` | +| R5 | benchmark 不影响评审 Skill | `without_skill` 变体仍安装 `judge.skills`,但不安装 `eval.skills` | +| R6 | 路径解析一致 | 本地 judge Skill 路径相对 Skill 根目录解析,与现有 `eval.skills` 约定一致 | +| R7 | 向后兼容 | 旧配置不需要修改,现有 `judge.criteria` 行为保持不变 | +| R8 | 原生 Skill 渐进式加载 | 实现必须调用 judge Agent adapter 的 `InstallSkill`;不得读取 Skill 文档并拼接到 judge prompt | +| R9 | 报告记录 judge Skill | JSON/HTML 报告能展示本次评审使用的 judge Skill 列表,JUnit 至少以 properties 形式暴露 | +| R10 | 强制使用 judge Skill | 当 `judge.skills` 非空时,`AgentJudge` 发送给 judge Agent 的 prompt 必须包含强制使用这些 Skill 的指令和 Skill 标识 | + +### 应该有 + +| ID | 需求 | 验收标准 | +| --- | --- | --- | +| S1 | 支持多个 judge Skills | `judge.skills` 可声明多个 Skill,并按配置顺序安装 | +| S2 | case 级覆盖 | case 内 `judge.skills` 可覆盖 eval 级 judge 配置 | +| S3 | 清晰诊断 | 安装失败时错误信息包含 judge Skill 路径和 judge 阶段上下文 | +| S4 | 文档更新 | 英文/中文 eval 写作指南与 skill-upper 参考文档包含示例 | +| S5 | Agent 差异文档化 | 文档说明 judge Skill 安装依赖具体 Agent 的 Skill 支持能力,不承诺所有 Agent 行为完全一致 | + +### 最好有 + +| ID | 需求 | 验收标准 | +| --- | --- | --- | +| N1 | 更丰富的报告元数据 | 报告可选记录 judge Skill 摘要哈希、安装目标和安装状态,不暴露敏感绝对路径 | +| N2 | 未来兼容 skill-only judge | 后续可在不破坏本设计的前提下,让 judge Skill 提供默认 criteria | + +## 提案 + +新增 `judge.skills` 字段,并采用复数形式,与顶层 `skills` 保持一致: + +```yaml +judge: + type: agent_judge + model: anthropic/claude-sonnet-4-6 + skills: + - source: local_path + path: evals/fixtures/judge-skill + criteria: + - "根据已安装的评审 Skill 中的 rubric 判断答案是否正确" +``` + +高层执行流程: + +1. 加载 `eval.yaml` 和 case YAML 时解析 `judge.skills`。 +2. 校验 `judge.skills` 只与 `agent_judge` 搭配使用。 +3. 执行 case 时,先按现有逻辑准备 runtime、安装主运行 Agent、MCP 和 `eval.skills`。 +4. 运行主 Agent,得到 `SessionResult`、workspace diff、transcript 等评审输入。 +5. 进入 judge 阶段,解析或创建 judge Agent。 +6. 在创建 `AgentJudge` 或调用 `Evaluate()` 前,通过 judge Agent adapter 的 `InstallSkill` 把合并后的 `judge.skills` 安装到 judge Agent。 +7. 构建评审提示时,如果 `judge.skills` 非空,加入强制使用已安装 judge Skill 的指令,不允许以普通 criteria-only 方式评分。 +8. 调用 judge Agent 运行评审提示,解析结构化 JSON 评分结果。 +9. 将本次评审使用的 judge Skill 元数据写入报告。 + +``` +┌─────────────────────────────────────────────────────────┐ +│ Case Runtime │ +│ │ +│ setupCaseEnvironment │ +│ ├─ install run agent │ +│ ├─ install MCP │ +│ └─ install eval.skills ───────────────┐ │ +│ │ │ +│ ┌─────────▼─────────┐ │ +│ │ Main Run Agent │ │ +│ │ runs evaluated │ │ +│ │ Skill │ │ +│ └─────────┬─────────┘ │ +│ │ │ +│ transcript/diff/output │ +│ │ │ +│ judge phase │ │ +│ ├─ resolve judge agent │ │ +│ ├─ install judge.skills ───────┐ │ │ +│ └─ AgentJudge.Evaluate │ │ │ +│ ▼ ▼ │ +│ ┌────────────────────┐ │ +│ │ Judge Agent │ │ +│ │ grades with rubric │ │ +│ │ Skill + criteria │ │ +│ └────────────────────┘ │ +└─────────────────────────────────────────────────────────┘ +``` + +### 用户场景速查 + +#### 场景 1:复用领域评审 rubric + +```yaml +schema_version: v1alpha1 + +skills: + - source: local_path + path: . + +judge: + type: agent_judge + model: anthropic/claude-sonnet-4-6 + skills: + - source: local_path + path: evals/fixtures/sql-judge-skill + criteria: + - "SQL 修改符合评审 Skill 中定义的安全与兼容性规则" + - "评审结论必须引用具体证据,而不是泛泛评价" +``` + +主运行 Agent 安装被测 Skill;judge Agent 安装 `sql-judge-skill`,用于加载数据库变更审核规则、反例和输出格式要求。 + +#### 场景 2:case 级评审 Skill 覆盖 + +```yaml +# evals/eval.yaml +judge: + type: agent_judge + model: anthropic/claude-sonnet-4-6 + skills: + - source: local_path + path: evals/fixtures/default-judge-skill + criteria: + - "根据默认评审 Skill 判断输出质量" +``` + +```yaml +# evals/cases/security-review.yaml +judge: + type: agent_judge + skills: + - source: local_path + path: evals/fixtures/security-judge-skill + criteria: + - "根据安全评审 Skill 判断是否发现高风险问题" +``` + +当 case 声明自己的 `judge.type` 时,case 级 judge 配置视为完整评审策略:`skills` 和 `criteria` 使用 case 级配置,`model`、`pass_threshold`、`timeout_seconds` 可继续沿用全局默认。 + +#### 场景 3:benchmark 中保持评分工具稳定 + +```yaml +benchmark: + enabled: true + +skills: + - source: local_path + path: . + +judge: + type: agent_judge + model: anthropic/claude-sonnet-4-6 + skills: + - source: local_path + path: evals/fixtures/judge-rubric + criteria: + - "根据 judge-rubric 判断输出是否满足验收标准" +``` + +benchmark 会分别执行: + +- `with_skill`:主 Agent 安装 `eval.skills`,judge Agent 安装 `judge.skills`。 +- `without_skill`:主 Agent 不安装 `eval.skills`,judge Agent 仍安装 `judge.skills`。 + +这样比较的是被测 Skill 对结果的影响,而不是评审工具是否存在。 + +### 注意事项、约束与说明 + +1. **继续要求 `criteria`**:本阶段 `judge.criteria` 仍是结构化评分项来源。judge Skill 可以承载长 rubric,但 YAML 中至少保留一条 criteria,用于确定评分结果数量和报告结构。 +2. **复用 `SkillRef`**:`judge.skills` 使用 `source`、`path`、`target`,不新增单数 `judge.skill`,避免两套配置语义并存。 +3. **必须依赖 Agent Skill 机制**:judge Skill 的价值在于 Agent 可以按需发现和加载 Skill 资源。实现不得把 Skill 目录内容整体读入 `criteria` 或 judge prompt。 +4. **Agent 安装方式允许差异**:Claude Code、Codex、Qoder CLI、自定义 Agent 的 Skill 安装位置和索引机制可以不同;skill-up 的职责是把同一个 `runtime.SkillConfig` 交给对应 adapter。 +5. **本地路径优先**:第一阶段按现有本地 Skill 安装能力实现;如果未来顶层 `skills` 支持 registry,`judge.skills` 可自然复用。 +6. **安装失败是 ERROR**:judge Skill 安装失败说明评审环境未准备好,应标记为 ERROR,而不是 FAIL。 +7. **不静默回退**:配置了 `judge.skills` 但无法安装时,不应继续用无 Skill 的 judge Agent 评分。 + +### 风险与缓解措施 + +| 风险 | 影响 | 概率 | 缓解措施 | +| --- | --- | --- | --- | +| judge Skill 意外安装到主 Agent | benchmark 结果被污染 | 中 | 安装逻辑只接收 `judgeAgent`,单独测试 run/judge agent 的安装记录 | +| `without_skill` 跳过 judge Skill | baseline 无法被同一 rubric 评审 | 中 | 安装 judge Skills 不受 `configName` 控制 | +| case/global 合并语义不清 | eval 作者难以预测使用哪个 judge Skill | 中 | 沿用现有 `MergeJudgeConfig` 的“case type 完整覆盖”规则,并在文档写明 | +| criteria 与 judge Skill rubric 冲突 | 评审结果不稳定 | 中 | 文档建议 criteria 写成稳定评分项,Skill 中放详细规则和输出约束 | +| 安装路径逃逸 Skill 根目录 | 可能读取非预期本地文件 | 低 | 复用或补强现有本地 Skill 路径校验,确保相对路径解析后仍在 Skill 根目录内 | +| 为兼容某些 Agent 而拼接 Skill 文档到 prompt | 失去渐进式加载,context 爆炸,行为偏离真实 Agent Skill 使用方式 | 中 | 明确禁止 prompt 拼接降级;adapter 不支持安装时直接 ERROR | +| 不同 Agent 的 Skill 发现机制不一致 | 同一 judge Skill 在不同引擎下行为有差异 | 中 | 保持 adapter 级安装实现和测试,报告记录 engine 与 judge Skill 信息 | +| 报告缺少 judge Skill 信息 | 评审依据不可审计,难以复现 | 中 | 在 EvalResult/报告生成链路新增 judge skill 元数据 | +| judge Skill 已安装但 Agent 未使用 | 评分仍按普通 criteria 执行,用户定义 rubric 失效 | 中 | `AgentJudge` prompt 强制要求使用已安装 judge Skill;单元测试断言 prompt,集成 fixture 验证行为 | + +## 设计细节 + +### 配置 Schema + +在 `internal/config/schema.go` 中扩展 `JudgeConfig`: + +```go +// JudgeConfig describes the evaluation strategy. +type JudgeConfig struct { + Type string `json:"type" yaml:"type"` + ScriptPath string `json:"script_path,omitempty" yaml:"script_path,omitempty"` + Model string `json:"model,omitempty" yaml:"model,omitempty"` + Criteria []string `json:"criteria,omitempty" yaml:"criteria,omitempty"` + Skills []SkillRef `json:"skills,omitempty" yaml:"skills,omitempty"` + + PassThreshold *float64 `json:"pass_threshold,omitempty" yaml:"pass_threshold,omitempty"` + TimeoutSeconds *int `json:"timeout_seconds,omitempty" yaml:"timeout_seconds,omitempty"` + Success []Rule `json:"success,omitempty" yaml:"success,omitempty"` + Failure []Rule `json:"failure,omitempty" yaml:"failure,omitempty"` +} +``` + +校验规则: + +1. `judge.skills` 非空时,`judge.type` 必须是 `agent_judge`。 +2. 每个 `judge.skills[*].source` 目前应为 `local_path` 或沿用顶层 `skills` 已支持的 source 值。 +3. `source: local_path` 时 `path` 必填,且不能是空白字符串。 +4. `target` 可选;语义与顶层 `skills[*].target` 一致。 +5. `agent_judge` 仍要求 `model` 和至少一条 `criteria`,除非后续另行修改 `AgentJudge` 评分协议。 +6. 依赖继承默认值的校验,尤其是 `agent_judge` 必填的 `model`,必须在 `judge.MergeJudgeConfig(global, caseLevel)` 得到有效 judge 配置后执行。raw case 校验可以继续检查只依赖 case 本身的约束,但不能仅因为 case 级 `agent_judge` 的 `model` 来自全局 judge 配置就拒绝该 case。 + +### 配置合并语义 + +沿用当前 `judge.MergeJudgeConfig(global, caseLevel)` 的设计: + +- 如果 case 没有声明 `judge.type`,使用全局 judge 配置,包括全局 `judge.skills`。 +- 如果 case 声明了 `judge.type`,case 级配置视为完整覆盖;`skills`、`criteria`、`success`、`failure`、`script_path` 等使用 case 级值。 +- `model`、`pass_threshold`、`timeout_seconds` 可继续从全局配置继承,因为现有逻辑已这样处理。 + +新增字段后的合并逻辑无需特殊“追加”行为。原因是 judge Skill 通常代表一套完整评分上下文,隐式合并全局和 case 级 Skill 容易造成 rubric 冲突。如果 eval 作者确实需要多个 Skill,应在 case 级 `judge.skills` 中显式列出全部 Skill。 + +### 路径解析与安装目标 + +本地 judge Skill 路径解析规则与顶层 `skills` 保持一致: + +```go +skillSourceDir := e.loader.SkillDir() +skillSource := filepath.Join(skillSourceDir, judgeSkillRef.Path) +skillCfg := runtime.SkillConfig{ + Source: skillSource, + Target: judgeSkillRef.Target, +} +``` + +实现时建议抽取一个小的共享 helper,例如: + +```go +func resolveSkillConfig(skillDir string, ref config.SkillRef) runtime.SkillConfig { + return runtime.SkillConfig{ + Source: filepath.Join(skillDir, ref.Path), + Target: ref.Target, + } +} +``` + +这样顶层 `eval.skills` 和 `judge.skills` 共享路径解析,减少未来 drift。 + +需要注意: + +- 相对路径以 Skill 根目录为基准,而不是当前工作目录或 `eval.yaml` 所在目录。 +- 安装顺序按 `judge.skills` 数组顺序执行。 +- 错误信息应包含 `judge.skills[i].path`,便于定位。 + +### Agent 安装适配与渐进式加载 + +不同 Agent 对 Skill 的约定可能不同: + +- Claude Code 可能有自己的 Skill 目录和索引规则。 +- Codex 可能有自己的 Skill 搜索、资源发现和加载约定。 +- Qoder CLI 可能有不同的安装命令、目标目录或 manifest 处理方式。 +- custom agent 可能通过 `InstallSkillCmd` 或自身协议实现安装。 + +因此,本提案不要求 evaluator 理解每种 Agent 的 Skill 内部结构。evaluator 只做三件事: + +1. 根据配置解析出 `runtime.SkillConfig{Source, Target}`。 +2. 在正确阶段调用最终 judge Agent 的 `InstallSkill(ctx, rt, skillCfg)`。 +3. 记录安装的 judge Skill 元数据,供报告和诊断使用。 + +具体安装方式由 agent adapter 负责: + +```go +type Agent interface { + Run(ctx context.Context, rt Runtime, opts ExecOptions, messages []Message) (*SessionResult, error) + Install(ctx context.Context, rt Runtime) error + InstallMCP(ctx context.Context, rt Runtime, cfg runtime.MCPConfig) error + InstallSkill(ctx context.Context, rt Runtime, skillCfg runtime.SkillConfig) error + CheckCredentials(ctx context.Context) error + Name() string +} +``` + +关键约束: + +- `InstallSkill` 必须尽量使用该 Agent 的原生 Skill 安装方式,包括目录布局、索引文件、manifest、缓存刷新或命令行安装入口。 +- evaluator 不读取 judge Skill 的 `SKILL.md`、`references/`、`assets/`,也不把这些文件拼入 judge prompt。 +- 如果某个 adapter 无法安装 Skill,应返回明确错误;配置了 `judge.skills` 时不允许静默退回到“把文档塞进 prompt”的模式。 +- 对自定义 Agent,继续沿用已有 `InstallSkillCmd` 能力;没有安装能力而又配置 `judge.skills` 时,应在评审阶段失败为 ERROR。 + +这样可以保留 Skill 的核心价值:Agent 在评审过程中根据任务需要渐进式加载相关说明和资源,而不是一次性消耗上下文窗口。 + +### 强制使用语义 + +仅安装 judge Skill 不代表 `agent_judge` 一定会使用它。为了让用户定义的 judge Skill 真正参与评分,`AgentJudge` 的 prompt 构建逻辑必须在 `judge.skills` 非空时加入一个明确的强制段,例如: + +```text +You MUST use the installed judge Skill(s) listed below as the authoritative +grading rubric before evaluating the case. Do not grade this case using only +the inline criteria. The inline criteria identify the result dimensions, while +the judge Skill(s) define the detailed rubric, constraints, and evidence rules. + +Installed judge Skill(s): +- evals/fixtures/judge-skill +``` + +这段指令只引用 Skill 标识和配置路径,不展开 Skill 文件内容。它的作用是触发 Agent 的 Skill 发现/选择机制,并把评分依据的优先级说清楚: + +1. `judge.skills` 提供详细 rubric、约束、风格指南和证据要求。 +2. `judge.criteria` 提供结构化评分维度和报告结果数量。 +3. 当二者存在冲突时,评审 prompt 应明确以 judge Skill 中的 rubric 为准,除非 criteria 定义了更具体的用例级验收项。 + +实现要求: + +- `AgentJudge.buildPrompt()` 或等价函数接收已解析的 judge Skill 元数据。 +- 当 `judge.skills` 非空时,prompt 中必须包含“必须使用已安装 judge Skill”的强制指令和每个 Skill 的稳定标识。 +- 当 `judge.skills` 为空时,prompt 保持现有 criteria-only 行为,确保向后兼容。 +- 不要求 evaluator 证明 Agent 内部实际加载了哪个文件;但必须让可观测输入明确要求使用 Skill,并通过集成 fixture 证明支持 Skill 的 Agent 会按该指令使用 Skill。 +- 报告中应区分“configured/installed judge skills”和未来可能支持的“agent-acknowledged used judge skills”。本提案至少要求前者。 + +### 评估器执行流程 + +在 `defaultEvaluator.newJudgeForCase()` 附近加入 judge Skill 安装步骤: + +```go +func (e *defaultEvaluator) newJudgeForCase( + ctx context.Context, + rt runtime.Runtime, + judgeCfg config.JudgeConfig, + runAgent agent.Agent, +) (judge.Judge, error) { + judgeCfg = resolveJudgeScriptPath(e.judgeScriptBaseDir(), judgeCfg) + + judgeAgent, err := e.resolveJudgeAgent(ctx, judgeCfg, runAgent) + if err != nil { + return nil, err + } + + if err := e.installJudgeSkills(ctx, rt, judgeCfg, judgeAgent); err != nil { + return nil, err + } + + j, err := judge.NewJudge(judgeCfg, judgeAgent, rt) + if err != nil { + return nil, fmt.Errorf("failed to create judge: %w", err) + } + return j, nil +} +``` + +`installJudgeSkills` 的核心规则: + +```go +func (e *defaultEvaluator) installJudgeSkills( + ctx context.Context, + rt runtime.Runtime, + judgeCfg config.JudgeConfig, + judgeAgent agent.Agent, +) error { + if judgeCfg.Type != "agent_judge" || len(judgeCfg.Skills) == 0 { + return nil + } + if e.loader == nil { + return errors.New("judge.skills requires a loader to resolve local paths") + } + + skillDir := e.loader.SkillDir() + for i, ref := range judgeCfg.Skills { + skillCfg := resolveSkillConfig(skillDir, ref) + if err := judgeAgent.InstallSkill(ctx, rt, skillCfg); err != nil { + return fmt.Errorf("failed to install judge skill judge.skills[%d].path=%q: %w", i, ref.Path, err) + } + logging.DebugContextf(ctx, "Evaluator: judge skill installed: %s", filepath.Base(skillCfg.Source)) + } + return nil +} +``` + +安装时机选择在 `resolveJudgeAgent()` 之后、`judge.NewJudge()` 之前,有三个好处: + +1. 已经拿到最终 judge Agent,不会误装到 run Agent。 +2. runtime 已经创建,主运行阶段输出和 workspace diff 已准备好。 +3. 如果安装失败,可以在进入 `AgentJudge.Evaluate()` 之前返回明确 ERROR。 + +### 隔离语义 + +| 配置 | 安装到主运行 Agent | 安装到 judge Agent | +| --- | --- | --- | +| `eval.skills` + `with_skill` | 是 | 否 | +| `eval.skills` + `without_skill` | 否 | 否 | +| `judge.skills` + `with_skill` | 否 | 是 | +| `judge.skills` + `without_skill` | 否 | 是 | + +如果当前实现中 judge Agent 复用 `runAgent` 实例,仍应按“逻辑角色”处理安装:`eval.skills` 的安装发生在主运行前,`judge.skills` 的安装发生在评审前。对于内置引擎,优先保持 `agent_judge` 使用独立 judge Agent;如果不得不复用实例,文档和测试必须确保 judge Skill 安装不会影响主运行阶段,因为主运行已经结束。 + +### 报告元数据 + +报告应记录本次评审实际使用的 judge Skill 信息,便于审计和复现。建议在 case 结果或 grading metadata 中增加只含非敏感信息的结构: + +```go +// JudgeSkillInfo describes a judge Skill used during agent_judge evaluation. +type JudgeSkillInfo struct { + Source string `json:"source,omitempty"` // e.g. local_path + Path string `json:"path,omitempty"` // config path, relative when configured that way + Target string `json:"target,omitempty"` + Name string `json:"name,omitempty"` // derived from path basename or Skill metadata +} +``` + +报告约定: + +- JSON 报告:在每个 case/config result 下包含 `judge_skills` 或等价字段。 +- HTML 报告:在 case 详情中展示 judge Skill 名称、配置路径和 target。 +- JUnit 报告:通过 testcase properties 暴露 `judge.skills.count` 和 `judge.skills..path` 等基础字段。 +- Anthropic `grading.json`:如果格式没有合适扩展点,可不写入正式 grading 字段,但应在 skill-up 自有 JSON/HTML 报告中保留。 + +安全与隐私约束: + +- 默认记录用户配置中的相对 `path`,不记录展开后的本机绝对路径。 +- 不记录 Skill 文件全文。 +- 如未来增加 hash,可记录 Skill 目录摘要用于复现,但应避免把大文件内容直接嵌入报告。 + +### 文档与模板更新 + +需要更新: + +- `docs/guide/writing-evals.md`:在 `judge: agent_judge` 小节加入 `skills` 示例和隔离说明。 +- `docs/zh/guide/writing-evals.md`:同步中文说明。 +- `skills/skill-upper/references/eval-yaml.md`:补充 `judge.skills` 字段。 +- `skills/skill-upper/references/judge-types.md`:说明何时使用 judge Skill,何时继续使用 inline criteria。 +- Agent 文档或自定义引擎文档:说明 judge Skill 安装依赖对应 adapter 的 `InstallSkill` 能力,不支持时不会拼接文档降级。 +- 相关模板:如果模板包含 `agent_judge` 示例,可加入注释性示例,但不默认启用,以免增加成本。 + +## 测试计划 + +### 单元测试 + +1. **配置加载** + - eval 级 `judge.skills` 可正确反序列化到 `JudgeConfig.Skills`。 + - case 级 `judge.skills` 可正确反序列化。 + - 多个 Skill 保持配置顺序。 + +2. **配置校验** + - `judge.type: agent_judge` + `judge.skills` 校验通过。 + - `judge.type: rule_based` + `judge.skills` 校验失败。 + - `judge.type: script` + `judge.skills` 校验失败。 + - `judge.skills[*].path` 为空时校验失败。 + - `agent_judge` 缺少 `criteria` 时保持现有失败行为。 + +3. **配置合并** + - case 不声明 `judge.type` 时继承全局 `judge.skills`。 + - case 声明 `judge.type` 且声明自己的 `judge.skills` 时,不追加全局 `judge.skills`。 + - case 声明 `judge.type` 但未声明 `model` 时,仍继承全局 `model`。 + +4. **评估器安装行为** + - mock run Agent 只收到 `eval.skills` 安装。 + - mock judge Agent 只收到 `judge.skills` 安装。 + - `without_skill` 变体不安装 `eval.skills`,但安装 `judge.skills`。 + - judge Skill 安装失败时,case 状态为 ERROR,且不调用 judge Agent `Run()`。 + - adapter 不支持 Skill 安装且配置 `judge.skills` 时,不拼接 Skill 文档到 prompt,而是返回 ERROR。 + +5. **强制使用 prompt** + - `judge.skills` 非空时,`AgentJudge` prompt 包含必须使用已安装 judge Skill 的指令。 + - prompt 包含每个 judge Skill 的稳定标识或配置路径。 + - prompt 不包含 judge Skill 文件正文。 + - `judge.skills` 为空时,prompt 与现有 criteria-only 行为保持兼容。 + +6. **路径解析** + - 相对路径基于 Skill 根目录解析。 + - `target` 正确传递到 `runtime.SkillConfig.Target`。 + +7. **报告元数据** + - JSON 报告包含本次使用的 judge Skill 列表。 + - HTML 报告能展示 judge Skill 名称和配置路径。 + - JUnit properties 包含 judge Skill 数量和路径。 + - 报告不包含 judge Skill 文件全文或本机绝对路径。 + +### 集成 / E2E 风格测试 + +新增一个 fixture,例如: + +```text +e2e/testdata/agent-judge-skill/ + SKILL.md + evals/ + eval.yaml + cases/ + uses-judge-skill.yaml + fixtures/ + judge-skill/ + SKILL.md +``` + +该 fixture 使用可控的 mock/custom agent,验证: + +- 主运行 Agent 不读取 judge Skill。 +- judge Agent 收到的评审提示明确要求使用已安装 judge Skill。 +- judge Agent 通过 Skill 机制读取 judge Skill 的规则,而不是从 prompt 正文直接获得规则。 +- 缺少 judge Skill 时同一 case 会失败或 ERROR,从而证明安装确实生效。 +- mock/custom agent 通过自身 Skill 发现机制加载 judge Skill,测试不依赖把 `SKILL.md` 内容拼进 prompt。 +- 生成的报告包含 judge Skill 元数据。 + +## 缺点 + +1. **配置表面积增加**:`judge` 下新增 `skills` 字段,需要文档解释它与顶层 `skills` 的区别。 +2. **评审可复现性依赖 Skill 内容**:rubric 从 YAML 转移到 Skill 后,review 时需要同时查看 Skill 文件。 +3. **安装成本增加**:每次 `agent_judge` 都可能多一次或多次 Skill 安装。 +4. **criteria 仍需保留**:第一阶段没有完全实现“Skill 独立定义所有评分项”,对部分用户来说仍需要一条简短 criteria。 +5. **跨 Agent 行为不完全一致**:不同 Agent 的 Skill 发现和加载机制本来就不同,同一 judge Skill 在不同引擎下可能有细微行为差异。 +6. **报告需要扩展**:JSON/HTML/JUnit 报告链路都要携带 judge Skill 元数据,增加少量实现工作。 + +## 替代方案 + +### 方案 A:新增单数 `judge.skill` + +```yaml +judge: + type: agent_judge + skill: + source: local_path + path: evals/fixtures/judge-skill +``` + +优点是对单个 Skill 场景更简洁。缺点是顶层已有 `skills` 数组,新增单数会制造两套语义;未来支持多个 judge Skills 时还要迁移。因此不采用。 + +### 方案 B:把 judge Skill 放入顶层 `skills` + +```yaml +skills: + - source: local_path + path: . + - source: local_path + path: evals/fixtures/judge-skill +``` + +优点是不改 schema。缺点是 judge Skill 会安装到主运行 Agent,污染被测能力,尤其破坏 benchmark `without_skill` 的解释性。因此不采用。 + +### 方案 C:只扩展 `criteria`,支持引用外部 Markdown + +```yaml +judge: + type: agent_judge + criteria_file: evals/fixtures/rubric.md +``` + +优点是实现简单。缺点是无法利用 Agent Skill 的既有约定、目录结构、资源文件、安装机制和渐进式加载能力。如果实现为读取 Markdown 后拼入 prompt,还会把长 rubric 一次性塞进 context,正好违背 issue 中希望使用 Skill 的核心动机。因此不作为本提案主方向。 + +### 方案 C2:安装失败时把 Skill 文档拼进 prompt + +这种方案看似提高兼容性,但会让不同 Agent 的 Skill 行为退化成普通长 prompt: + +- 失去按需加载能力。 +- 容易把 `references/` 和资产说明一次性塞爆上下文。 +- 无法验证 judge Skill 在真实 Agent Skill 机制下是否可用。 +- 报告里很难说明到底是“安装了 Skill”还是“拼接了文档”。 + +因此本提案明确不采用该降级方案。 + +### 方案 D:允许 judge Skill 完全替代 criteria + +```yaml +judge: + type: agent_judge + skills: + - source: local_path + path: evals/fixtures/judge-skill +``` + +这是有价值的未来方向,但需要调整 `AgentJudge` 的 prompt 和 JSON 结果解析:当前报告结构依赖 criteria 数量来校验评审结果。本提案先保持 `criteria` 必填,降低实现风险。 + +## 所需基础设施 + +不需要新增外部服务或第三方依赖。实现仅复用现有配置加载、runtime Skill 安装、Agent 接口和测试基础设施。 + +可能需要的本地测试资产: + +- 一个 judge Skill fixture。 +- 一个记录 `InstallSkill` 调用的 mock judge Agent。 +- 一个用于集成验证的 custom/mock agent fixture。 + +## 升级与迁移策略 + +该变更向后兼容: + +- 现有 `agent_judge` 配置无需修改。 +- 不配置 `judge.skills` 时行为完全不变。 +- 顶层 `skills` 语义不变。 +- benchmark 输出语义不变,只是允许评审侧拥有稳定的专用 rubric Skill。 + +推荐迁移路径: + +1. 先把长篇 judge rubric 提取到 `evals/fixtures/-judge-skill/SKILL.md`。 +2. 在 `judge.skills` 中引用该路径。 +3. 将 `judge.criteria` 收敛为少量稳定评分项,例如“根据已安装评审 Skill 判断是否满足安全规则”。 +4. 运行 `skill-up validate` 和至少一个小规模 eval,确认 judge Agent 能读取评审 Skill。 diff --git a/skills/skill-upper/assets/case.yaml.tmpl b/skills/skill-upper/assets/case.yaml.tmpl index c7c50cf0..c8b225eb 100644 --- a/skills/skill-upper/assets/case.yaml.tmpl +++ b/skills/skill-upper/assets/case.yaml.tmpl @@ -16,8 +16,11 @@ input: # post_condition: # must_contain_any: ["keywordA", "keywordB"] # on_fail: skip_remaining # or fail +# capture: +# - variable: var_name +# pattern: "(?Ppattern)" # - role: user -# content: "Second request" +# content: "Follow-up using {{var_name}}" # context: # Optional: initialize workspace # repo_fixture: fixtures/repos/sample-project @@ -59,3 +62,20 @@ judge: # - "Output correctly identifies user intent" # - "Does not fabricate non-existent fields" # pass_threshold: 0.7 + +# Per-turn assertions (use with input.turns): +# judge: +# type: rule_based +# success: +# - turn_response_contains: +# turn: 1 +# contains_all: ["expected_keyword"] +# - turn_response_not_contains: +# turn: 2 +# not_contains: ["forbidden_word"] +# - tool_called_in_turn: +# turn: 1 +# name: write_file +# - tool_not_called_in_turn: +# turn: 2 +# name: delete_file diff --git a/skills/skill-upper/references/case-yaml.md b/skills/skill-upper/references/case-yaml.md index cc6164e0..07ccd247 100644 --- a/skills/skill-upper/references/case-yaml.md +++ b/skills/skill-upper/references/case-yaml.md @@ -47,12 +47,43 @@ input: post_condition: must_contain_any: ["Research", "分析"] on_fail: skip_remaining # 或 fail + capture: + - variable: phase + pattern: "(?PResearch|Implementation)" - role: user content: "跳过 Research,直接帮我写代码" ``` `post_condition`:每轮结束后检查输出。`on_fail: skip_remaining` 标为 SKIP,`fail` 直接 FAIL 整个用例。 +`capture`:从响应中捕获值,用于后续轮次的 `{{variable}}` 模板替换。 +  • 提取器:`pattern`(正则)或 `jsonpath`,必须且仅指定一个 +  • 优先使用 `(?P...)` 命名组 +  • 未匹配/空值 → 用例进入 ERROR 状态 +  • 作用域仅限当前用例执行 + +### 按轮次 Judge 断言 + +```yaml +judge: + type: rule_based + success: + - turn_response_contains: + turn: 2 + contains_any: ["必须完成", "不能跳过", "Research"] + - turn_response_not_contains: + turn: 2 + not_contains: ["LGTM"] + - tool_called_in_turn: + turn: 1 + name: write_file + - tool_not_called_in_turn: + turn: 2 + name: delete_file +``` + +仅 `status=completed` 的轮次可被断言;引用不存在或未完成的轮次会导致断言失败。 + ## context — 初始化工作区 ```yaml diff --git a/skills/skill-upper/references/eval-yaml.md b/skills/skill-upper/references/eval-yaml.md index efc3034b..43815f1b 100644 --- a/skills/skill-upper/references/eval-yaml.md +++ b/skills/skill-upper/references/eval-yaml.md @@ -42,6 +42,15 @@ cases: max_retries: 1 retry_on: [timeout, error] +judge: + type: agent_judge + model: anthropic/claude-sonnet-4-6 + skills: # 可选:仅安装给 judge agent 的评分 Skill + - source: local_path + path: evals/fixtures/judge-rubric + criteria: + - "按 judge-rubric 中的细则判断输出是否满足要求" + benchmark: enabled: false @@ -54,6 +63,8 @@ report: `collect_artifacts`(`cases.defaults` 级,或单个 `case.yaml` 内追加)用 [doublestar](https://github.com/bmatcuk/doublestar) glob(`*` 单层、`**` 跨目录)声明要采集的 workspace 文件。无论 Agent 成功/失败/超时,命中文件都会保留相对路径下载到 `///outputs/workspace/`。两层按并集去重合并。它与 `report.artifacts`(产物*类型*)、`agent_judge` 的 git diff(字符串)正交。 +`judge.skills` 仅支持 `judge.type: agent_judge`,用于给 judge agent 安装可复用的评分 Rubric Skill。它不会安装到主运行 agent;顶层 `skills` 也不会自动安装到 judge。benchmark 下 `with_skill` / `without_skill` 都会安装 judge Skills,因为它们属于评分工具。路径相对 Skill 根目录解析,安装依赖具体 Agent adapter 的原生 Skill 支持;不要把 Skill 文件内容复制进 `criteria`。 + ## 运行环境 | type | 适用场景 | 说明 | diff --git a/skills/skill-upper/references/judge-types.md b/skills/skill-upper/references/judge-types.md index 84357a56..35d4a820 100644 --- a/skills/skill-upper/references/judge-types.md +++ b/skills/skill-upper/references/judge-types.md @@ -45,8 +45,11 @@ judge: judge: type: agent_judge model: anthropic/claude-sonnet-4-6 + skills: + - source: local_path + path: evals/fixtures/judge-rubric criteria: - - "输出中识别了真实存在的 bug,并给出了准确位置" + - "输出中识别了真实存在的 bug,并符合 judge-rubric 中的评分细则" - "没有将正确代码误报为 bug" - "建议具有可操作性,不是泛泛而谈" pass_threshold: 0.7 @@ -57,6 +60,8 @@ judge: - 会额外消耗 token,慢且贵 - criteria 尽量具体、可验证 - 能拆出确定性条件时先用 `rule_based` / `expect` 挡一道 +- 需要长 Rubric、领域规则、复用评分规范时,把它们放进 `judge.skills` +- `judge.skills` 只会安装给 judge agent,不会污染被测 run agent;安装依赖具体 Agent adapter 的 Skill 支持,不会回退为 prompt 拼接 ## script — 自定义脚本