feat(judge): support judge-specific skills#141
Draft
roark47 wants to merge 22 commits into
Draft
Conversation
Add schema types and validation rules for multi-turn conversation evaluation: Schema changes (internal/config/schema.go): - Turn.Capture []CaptureRule for variable extraction from responses - Turn.TimeoutSeconds for per-turn timeout control - PostCondition.MustContainAll and MustNotContain fields - CaptureRule type (Variable, Pattern, JSONPath) - TurnResponseContainsRule, TurnResponseNotContainsRule - ToolCalledInTurnRule, ToolNotCalledInTurnRule - Rule struct extended with 4 per-turn assertion fields Validation changes (internal/config/validator.go): - Reject ambiguous input.prompt + input.turns combination - Enforce input.turns[*].role must be "user" - Validate post_condition.on_fail enum (fail, skip_remaining) - Require at least one check in post_condition - Validate capture variable identifier pattern - Enforce exactly one extractor per capture rule - Compile-time regex validation for capture patterns - Per-turn judge rules: turn number range checks - Required fields for turn_response_contains/not_contains - Required name for tool_called_in_turn/tool_not_called_in_turn All existing tests pass unchanged (backward compatible).
Introduce SessionResumer optional interface and SessionResult.SessionID field to support multi-turn conversation evaluation: Interface (internal/agent/agent.go): - SessionResult.SessionID string field for session tracking across turns - SessionResumer interface with RunTurn method - Compile-time satisfaction checks for ClaudeCodeAgent, QoderCLIAgent ClaudeCodeAgent (internal/agent/claude_code.go): - Run populates SessionResult.SessionID from generated UUID - RunTurn: empty sessionID delegates to Run (first turn) - RunTurn: non-empty sessionID uses --resume flag - buildClaudeResumeCmd for session resume commands - Extract claudeRunEnvVars and handleClaudeRunResult helpers QoderCLIAgent (internal/agent/qodercli.go): - buildSessionResult extracts SessionID from session JSONL filename - RunTurn: empty sessionID delegates to Run (first turn) - RunTurn: non-empty sessionID uses -r flag - buildQoderResumeCmd for session resume commands Utilities (internal/agent/session_lookup.go): - extractSessionIDFromPath strips directory and extension All existing tests pass unchanged (backward compatible).
Implement the evaluator multi-turn engine that orchestrates per-turn
agent execution with post-condition gates, variable capture, and
template substitution:
Types (internal/evaluator/multiturn.go):
- TurnStatus: completed, skipped, failed, error
- TurnResult: per-turn outcome with response, transcript, captured vars
- multiTurnState: mutable state for the multi-turn loop
Engine (internal/evaluator/multiturn.go):
- executeMultiTurn: main loop — template substitution, agent invocation,
post-condition evaluation, variable capture, result aggregation
- executeSingleTurn: per-turn timeout + resumer invocation
- buildTurnResult: construct TurnResult from agent SessionResult
- handlePostCondition: on_fail=fail (abort) / skip_remaining (mark rest)
- evaluatePostCondition: must_contain_all/any, must_not_contain checks
- captureVariables: regex named group (?P<value>...) or first group
- substituteTemplate: {{variable}} replacement with unresolved error
- buildAggregateResult: merge per-turn metrics into final SessionResult
- multiTurnStatus: derive case status from turn outcomes
Integration (internal/evaluator/evaluator.go):
- EvalResult.TurnResults []TurnResult field for per-turn data
- executeCaseOnce branches to multi-turn when len(turns)>1 AND agent
implements SessionResumer; non-resumer agents fall through to the
existing batch Run path (backward compatible)
Tests (internal/evaluator/multiturn_test.go):
- TestSubstituteTemplate (6 cases)
- TestEvaluatePostCondition (8 cases)
- TestCaptureVariables (6 cases)
- TestMultiTurnStatus (5 cases)
- TestExecuteMultiTurn_HappyPath
- TestExecuteMultiTurn_PostConditionFail
- TestExecuteMultiTurn_PostConditionSkipRemaining
- TestExecuteMultiTurn_TemplateSubstitution
- TestExecuteMultiTurn_AgentError
- TestExecuteMultiTurn_NoResumer
- TestExecuteMultiTurn_UnresolvedTemplate
Post Conditions: - extractCaptureValue rejects multiple unnamed capture groups (requires exactly one or a named group 'value') Capture Variables: - Add comprehensive test coverage for edge cases: multiple unnamed groups error, case-sensitivity in post_condition matchers, named group priority Tests: - TestCaptureVariables_EdgeCases: multi-group error, named+unnamed priority - TestPostConditionCaseSensitivity: explicit case-sensitive assertion - TestExtractCaptureValue: unit tests for the extraction helper - Split long test functions to satisfy funlen linter (80 stmt limit) Lint: - Replace fmt.Errorf (no formatting) with errors.New - gofumpt formatting fixes
Judge Input: - Add InputTurnResult type with TurnNumber, Content, Response, Transcript, Status, and Reason fields - Add TurnResults []InputTurnResult field to judge.Input Per-turn rule evaluators (internal/judge/rule_based.go): - turn_response_contains: check contains_all/contains_any on a specific turn's response - turn_response_not_contains: check forbidden keywords absence - tool_called_in_turn: verify tool usage with optional partial args - tool_not_called_in_turn: verify tool absence in a turn - Shared lookupTurn helper: validates turn existence and completed status before content assertions Evaluator wiring (internal/evaluator/evaluator.go): - Add toJudgeTurnResults converter from evaluator.TurnResult to judge.InputTurnResult - Pass TurnResults in judge.Input within evaluateCaseSession Tests: - 16 new test cases covering all 4 per-turn rules: positive/negative matching, missing turns, skipped turns, failed turns, args matching - Table-driven patterns to satisfy dupl linter
CaseResult (internal/report/reporter.go): - Add TurnResults []CaseTurnResult field (omitempty for backward compat) - Define CaseTurnResult with TurnNumber, Content, Response, Status, Reason JSON report: - turn_results automatically serialized in result.json/report.json - Omitted for single-turn cases (nil slice + omitempty) HTML report (internal/report/html.go): - Add embeddedTurn type and TurnResults field to embeddedCase - caseTurnResultsToEmbedded converts report types to embedded JS data JUnit report (internal/report/junit.go): - buildFailureBody appends non-completed turn status/reason lines - Turn-scoped assertion texts already contain turn numbers for CI Runner wiring (internal/runner/runner.go): - evalResultToCaseResult passes TurnResults via evalTurnResultsToReport - Converter maps evaluator.TurnResult to report.CaseTurnResult
writing-evals.md (EN + ZH): - New 'Multi-turn conversations' section covering input.prompt vs input.turns decision matrix, post_condition gate semantics, capture/template variable syntax, agent support matrix, and per-turn judge assertion examples. e2e/testdata/multi-turn-conversation: - phase-gate-skip-attempt.yaml: convert fake single-prompt to real input.turns with post_condition + turn_response_contains assertions. - multi-turn-refinement.yaml: split into 2 real turns with capture and per-turn assertions. skills/skill-upper: - references/case-yaml.md: expand multi-turn section with capture field reference and per-turn judge assertion examples. - assets/case.yaml.tmpl: add capture and per-turn judge commented examples. CHANGELOG.md: - Add [Unreleased] entries for multi-turn conversation feature, per-turn assertions, session resumption, and report TurnResults.
CodexAgent now implements the SessionResumer interface, enabling native multi-turn conversation support via `codex resume <thread-id>`. codex.go: - Add RunTurn method: first turn (sessionID=="") delegates to Run; subsequent turns build a `codex resume --json` command with the thread ID obtained from the first turn's event stream. - Set SessionResult.SessionID = streamParsed.threadID in buildSessionResult so the evaluator can pass it forward between turns. - Add buildCodexResumeCmdWithLastMessage helper (mirrors the exec variant but uses `codex resume` subcommand with sessionID positional arg). codex_test.go: - TestCodexRunTurn_FirstTurnDelegatesToRun: verifies first turn uses `codex exec` and extracts thread ID into SessionID. - TestCodexRunTurn_ResumeUsesCorrectCommand: verifies subsequent turns use `codex resume` with the session ID and instruction. - TestBuildCodexResumeCmdWithLastMessage: unit test for command builder. Docs: update agent support matrix in writing-evals.md (EN+ZH) and CHANGELOG.md to reflect Codex multi-turn support.
The top-level 'codex resume' command is a TUI-only subcommand that does not support --json, --skip-git-repo-check, or --output-last-message. Switch to 'codex exec resume' which is the non-interactive equivalent supporting all required flags. Remove the --sandbox flag from resume commands since the session inherits sandbox settings from its initial creation; use --dangerously-bypass-approvals-and-sandbox to ensure non-interactive execution.
When assertions pass, the evidence field now includes the actual keywords that were checked, making it possible to audit results without referring back to the case YAML. - turn_response_contains: evidence shows contains_all/contains_any lists - turn_response_not_contains: evidence shows the forbidden keywords list - output_contains: evidence shows all/any/not keyword lists
…lure multiTurnStatus previously treated all TurnFailed results as hard failures, skipping judge execution entirely. This broke the on_fail=skip_remaining contract which is designed to skip subsequent turns but still run the judge on whatever data was collected. Distinguish the two modes by checking whether TurnSkipped entries follow the failed turn — if so, it is a skip_remaining case and the evaluator proceeds to judge evaluation.
- Show per-turn Prompt/Response cards when turn_results is available, replacing the previous single prompt/response view - Synthesize FORMAL GRADES from turn post-condition failures when the judge was not executed (grading is null but turn_results has failures) - Hide the top-level Prompt section for multi-turn cases to avoid showing only the first turn's content
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
judge.skillstoagent_judgeconfiguration and validation.without_skillruns.Impact
This lets eval authors keep long or reusable grading rubrics in judge-only Skills while preserving the measured Skill-under-test environment. Existing configs without
judge.skillskeep their previous behavior.Validation
make fmtGOCACHE=/Users/kongtang/Code/skill-up-proposal-0002/.cache/go-build make verifyGOCACHE=/Users/kongtang/Code/skill-up-proposal-0002/.cache/go-build make testNote:
gh auth statusreports expired local tokens, so the PR was created through the GitHub connector after pushing the branch with git.