feat(judge): support judge-specific skills#141
Conversation
Add schema types and validation rules for multi-turn conversation evaluation: Schema changes (internal/config/schema.go): - Turn.Capture []CaptureRule for variable extraction from responses - Turn.TimeoutSeconds for per-turn timeout control - PostCondition.MustContainAll and MustNotContain fields - CaptureRule type (Variable, Pattern, JSONPath) - TurnResponseContainsRule, TurnResponseNotContainsRule - ToolCalledInTurnRule, ToolNotCalledInTurnRule - Rule struct extended with 4 per-turn assertion fields Validation changes (internal/config/validator.go): - Reject ambiguous input.prompt + input.turns combination - Enforce input.turns[*].role must be "user" - Validate post_condition.on_fail enum (fail, skip_remaining) - Require at least one check in post_condition - Validate capture variable identifier pattern - Enforce exactly one extractor per capture rule - Compile-time regex validation for capture patterns - Per-turn judge rules: turn number range checks - Required fields for turn_response_contains/not_contains - Required name for tool_called_in_turn/tool_not_called_in_turn All existing tests pass unchanged (backward compatible).
Introduce SessionResumer optional interface and SessionResult.SessionID field to support multi-turn conversation evaluation: Interface (internal/agent/agent.go): - SessionResult.SessionID string field for session tracking across turns - SessionResumer interface with RunTurn method - Compile-time satisfaction checks for ClaudeCodeAgent, QoderCLIAgent ClaudeCodeAgent (internal/agent/claude_code.go): - Run populates SessionResult.SessionID from generated UUID - RunTurn: empty sessionID delegates to Run (first turn) - RunTurn: non-empty sessionID uses --resume flag - buildClaudeResumeCmd for session resume commands - Extract claudeRunEnvVars and handleClaudeRunResult helpers QoderCLIAgent (internal/agent/qodercli.go): - buildSessionResult extracts SessionID from session JSONL filename - RunTurn: empty sessionID delegates to Run (first turn) - RunTurn: non-empty sessionID uses -r flag - buildQoderResumeCmd for session resume commands Utilities (internal/agent/session_lookup.go): - extractSessionIDFromPath strips directory and extension All existing tests pass unchanged (backward compatible).
Implement the evaluator multi-turn engine that orchestrates per-turn
agent execution with post-condition gates, variable capture, and
template substitution:
Types (internal/evaluator/multiturn.go):
- TurnStatus: completed, skipped, failed, error
- TurnResult: per-turn outcome with response, transcript, captured vars
- multiTurnState: mutable state for the multi-turn loop
Engine (internal/evaluator/multiturn.go):
- executeMultiTurn: main loop — template substitution, agent invocation,
post-condition evaluation, variable capture, result aggregation
- executeSingleTurn: per-turn timeout + resumer invocation
- buildTurnResult: construct TurnResult from agent SessionResult
- handlePostCondition: on_fail=fail (abort) / skip_remaining (mark rest)
- evaluatePostCondition: must_contain_all/any, must_not_contain checks
- captureVariables: regex named group (?P<value>...) or first group
- substituteTemplate: {{variable}} replacement with unresolved error
- buildAggregateResult: merge per-turn metrics into final SessionResult
- multiTurnStatus: derive case status from turn outcomes
Integration (internal/evaluator/evaluator.go):
- EvalResult.TurnResults []TurnResult field for per-turn data
- executeCaseOnce branches to multi-turn when len(turns)>1 AND agent
implements SessionResumer; non-resumer agents fall through to the
existing batch Run path (backward compatible)
Tests (internal/evaluator/multiturn_test.go):
- TestSubstituteTemplate (6 cases)
- TestEvaluatePostCondition (8 cases)
- TestCaptureVariables (6 cases)
- TestMultiTurnStatus (5 cases)
- TestExecuteMultiTurn_HappyPath
- TestExecuteMultiTurn_PostConditionFail
- TestExecuteMultiTurn_PostConditionSkipRemaining
- TestExecuteMultiTurn_TemplateSubstitution
- TestExecuteMultiTurn_AgentError
- TestExecuteMultiTurn_NoResumer
- TestExecuteMultiTurn_UnresolvedTemplate
Post Conditions: - extractCaptureValue rejects multiple unnamed capture groups (requires exactly one or a named group 'value') Capture Variables: - Add comprehensive test coverage for edge cases: multiple unnamed groups error, case-sensitivity in post_condition matchers, named group priority Tests: - TestCaptureVariables_EdgeCases: multi-group error, named+unnamed priority - TestPostConditionCaseSensitivity: explicit case-sensitive assertion - TestExtractCaptureValue: unit tests for the extraction helper - Split long test functions to satisfy funlen linter (80 stmt limit) Lint: - Replace fmt.Errorf (no formatting) with errors.New - gofumpt formatting fixes
Judge Input: - Add InputTurnResult type with TurnNumber, Content, Response, Transcript, Status, and Reason fields - Add TurnResults []InputTurnResult field to judge.Input Per-turn rule evaluators (internal/judge/rule_based.go): - turn_response_contains: check contains_all/contains_any on a specific turn's response - turn_response_not_contains: check forbidden keywords absence - tool_called_in_turn: verify tool usage with optional partial args - tool_not_called_in_turn: verify tool absence in a turn - Shared lookupTurn helper: validates turn existence and completed status before content assertions Evaluator wiring (internal/evaluator/evaluator.go): - Add toJudgeTurnResults converter from evaluator.TurnResult to judge.InputTurnResult - Pass TurnResults in judge.Input within evaluateCaseSession Tests: - 16 new test cases covering all 4 per-turn rules: positive/negative matching, missing turns, skipped turns, failed turns, args matching - Table-driven patterns to satisfy dupl linter
CaseResult (internal/report/reporter.go): - Add TurnResults []CaseTurnResult field (omitempty for backward compat) - Define CaseTurnResult with TurnNumber, Content, Response, Status, Reason JSON report: - turn_results automatically serialized in result.json/report.json - Omitted for single-turn cases (nil slice + omitempty) HTML report (internal/report/html.go): - Add embeddedTurn type and TurnResults field to embeddedCase - caseTurnResultsToEmbedded converts report types to embedded JS data JUnit report (internal/report/junit.go): - buildFailureBody appends non-completed turn status/reason lines - Turn-scoped assertion texts already contain turn numbers for CI Runner wiring (internal/runner/runner.go): - evalResultToCaseResult passes TurnResults via evalTurnResultsToReport - Converter maps evaluator.TurnResult to report.CaseTurnResult
writing-evals.md (EN + ZH): - New 'Multi-turn conversations' section covering input.prompt vs input.turns decision matrix, post_condition gate semantics, capture/template variable syntax, agent support matrix, and per-turn judge assertion examples. e2e/testdata/multi-turn-conversation: - phase-gate-skip-attempt.yaml: convert fake single-prompt to real input.turns with post_condition + turn_response_contains assertions. - multi-turn-refinement.yaml: split into 2 real turns with capture and per-turn assertions. skills/skill-upper: - references/case-yaml.md: expand multi-turn section with capture field reference and per-turn judge assertion examples. - assets/case.yaml.tmpl: add capture and per-turn judge commented examples. CHANGELOG.md: - Add [Unreleased] entries for multi-turn conversation feature, per-turn assertions, session resumption, and report TurnResults.
CodexAgent now implements the SessionResumer interface, enabling native multi-turn conversation support via `codex resume <thread-id>`. codex.go: - Add RunTurn method: first turn (sessionID=="") delegates to Run; subsequent turns build a `codex resume --json` command with the thread ID obtained from the first turn's event stream. - Set SessionResult.SessionID = streamParsed.threadID in buildSessionResult so the evaluator can pass it forward between turns. - Add buildCodexResumeCmdWithLastMessage helper (mirrors the exec variant but uses `codex resume` subcommand with sessionID positional arg). codex_test.go: - TestCodexRunTurn_FirstTurnDelegatesToRun: verifies first turn uses `codex exec` and extracts thread ID into SessionID. - TestCodexRunTurn_ResumeUsesCorrectCommand: verifies subsequent turns use `codex resume` with the session ID and instruction. - TestBuildCodexResumeCmdWithLastMessage: unit test for command builder. Docs: update agent support matrix in writing-evals.md (EN+ZH) and CHANGELOG.md to reflect Codex multi-turn support.
The top-level 'codex resume' command is a TUI-only subcommand that does not support --json, --skip-git-repo-check, or --output-last-message. Switch to 'codex exec resume' which is the non-interactive equivalent supporting all required flags. Remove the --sandbox flag from resume commands since the session inherits sandbox settings from its initial creation; use --dangerously-bypass-approvals-and-sandbox to ensure non-interactive execution.
When assertions pass, the evidence field now includes the actual keywords that were checked, making it possible to audit results without referring back to the case YAML. - turn_response_contains: evidence shows contains_all/contains_any lists - turn_response_not_contains: evidence shows the forbidden keywords list - output_contains: evidence shows all/any/not keyword lists
…lure multiTurnStatus previously treated all TurnFailed results as hard failures, skipping judge execution entirely. This broke the on_fail=skip_remaining contract which is designed to skip subsequent turns but still run the judge on whatever data was collected. Distinguish the two modes by checking whether TurnSkipped entries follow the failed turn — if so, it is a skip_remaining case and the evaluator proceeds to judge evaluation.
- Show per-turn Prompt/Response cards when turn_results is available, replacing the previous single prompt/response view - Synthesize FORMAL GRADES from turn post-condition failures when the judge was not executed (grading is null but turn_results has failures) - Hide the top-level Prompt section for multi-turn cases to avoid showing only the first turn's content
There was a problem hiding this comment.
Pull request overview
This PR adds judge-only Skills support for agent_judge, allowing eval authors to install reusable grading rubrics on the judge agent (without polluting the run agent), and to surface judge Skill metadata in generated reports.
Changes:
- Extend the eval schema + validation to support
judge.skillsand validate effective judge requirements after applying eval-level defaults. - Install judge Skills on the judge agent before grading and require rubric usage via prompt instructions (without inlining Skill contents).
- Expose judge Skill metadata in JSON/HTML/JUnit reports and update docs + tests accordingly.
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| skills/skill-upper/references/judge-types.md | Documents judge.skills usage for agent_judge in skill-upper references. |
| skills/skill-upper/references/eval-yaml.md | Adds judge.skills to eval.yaml reference docs and clarifies benchmark behavior. |
| internal/runner/runner.go | Plumbs JudgeSkills from evaluator results into report case results. |
| internal/report/reporter.go | Adds judge_skills to CaseResult JSON model. |
| internal/report/reporter_test.go | Updates report tests to assert judge skill metadata is preserved. |
| internal/report/junit.go | Adds JUnit <properties> emission for judge skill metadata. |
| internal/report/html.go | Embeds judge_skills into HTML report data model. |
| internal/platform/platform_test.go | Minor test control-flow tweak (explicit return after fatal). |
| internal/judge/judge_skill.go | Introduces SkillInfo and conversion from config skill refs for reporting/prompt metadata. |
| internal/judge/helpers_test.go | Captures last judge prompt messages in mock agent to assert prompt content. |
| internal/judge/factory.go | Passes judge skill metadata into AgentJudge construction. |
| internal/judge/factory_test.go | Adds coverage for judge skill propagation and merge semantics. |
| internal/judge/agent_judge.go | Adds judge-skill prompt requirements and stores judge skill metadata on AgentJudge. |
| internal/judge/agent_judge_test.go | Adds tests asserting mandatory skill-use instructions and prompt includes skill identifiers. |
| internal/evaluator/evaluator.go | Installs judge Skills on the judge agent and records judge skill metadata in results. |
| internal/evaluator/evaluator_test.go | Adds tests ensuring judge Skills install on judge agent only and install failures are surfaced. |
| internal/config/validator.go | Adds judge.skills validation + effective-judge validation after applying eval defaults. |
| internal/config/validator_test.go | Updates/extends validator tests for judge.skills and effective judge inheritance. |
| internal/config/schema.go | Adds Skills []SkillRef to JudgeConfig. |
| internal/config/schema_test.go | Minor test control-flow tweak (explicit return after fatal). |
| internal/config/loader_test.go | Adds loader coverage for judge.skills parsing. |
| internal/cli/run.go | Validates filtered case subsets using eval-level judge defaults (effective validation). |
| internal/cli/judge_debug.go | Adds lint suppression comment for debug JSON unmarshalling. |
| internal/cli/judge_debug_test.go | Adds lint suppression comment + minor control-flow tweak after fatal. |
| internal/agent/custom_test.go | Minor test control-flow tweak (explicit return after fatal). |
| internal/agent/codex_test.go | Minor test control-flow tweaks (explicit returns after fatal). |
| e2e/agent_test.go | Increases an e2e case timeout. |
| docs/zh/guide/writing-evals.md | Documents judge.skills in the Chinese guide. |
| docs/guide/writing-evals.md | Documents judge.skills in the English guide. |
| docs/design/custom-engine.md | Notes judge-skill installation expectations for custom engines. |
| CHANGELOG.md | Records the new judge.skills feature in release notes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ea3dc5c533
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This reverts commit e99ffe0.
Summary
judge.skillstoagent_judgeconfiguration and validation.without_skillruns.Impact
This lets eval authors keep long or reusable grading rubrics in judge-only Skills while preserving the measured Skill-under-test environment. Existing configs without
judge.skillskeep their previous behavior.Validation
make fmtGOCACHE=/Users/kongtang/Code/skill-up-proposal-0002/.cache/go-build make verifyGOCACHE=/Users/kongtang/Code/skill-up-proposal-0002/.cache/go-build make testNote:
gh auth statusreports expired local tokens, so the PR was created through the GitHub connector after pushing the branch with git.