Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
a93ce15
feat(config): add multi-turn conversation schema fields and validation
roark47 Jul 6, 2026
bdbd453
feat(agent): add SessionResumer interface for multi-turn conversation
roark47 Jul 6, 2026
5037fb9
feat(evaluator): add multi-turn conversation execution engine
roark47 Jul 6, 2026
009979c
feat(evaluator): strengthen post_condition and capture semantics
roark47 Jul 6, 2026
de841a9
feat(judge): add per-turn assertion rules for multi-turn evaluation
roark47 Jul 6, 2026
96a3422
feat(report): add per-turn results to report output formats
roark47 Jul 6, 2026
fd9ac51
docs: add multi-turn conversation documentation and update e2e examples
roark47 Jul 6, 2026
8a37fd4
feat(agent): implement SessionResumer for Codex engine
roark47 Jul 6, 2026
ab3dfa5
fix(agent): use codex exec resume for non-interactive session resumption
roark47 Jul 7, 2026
ca01982
fix(judge): include rule keywords in assertion evidence
roark47 Jul 7, 2026
1028b09
fix(evaluator): proceed to judge on skip_remaining post-condition fai…
roark47 Jul 7, 2026
5605b45
fix(report): render multi-turn conversation and failure reasons in HTML
roark47 Jul 7, 2026
0403c16
chore: ignore local verification project directory
roark47 Jul 7, 2026
8fd2360
chore: Update changelog
roark47 Jul 7, 2026
93c6907
fix(evaluator): correct multi-turn execution accounting
roark47 Jul 7, 2026
f972d95
fix: address copilot multi-turn review feedback
roark47 Jul 7, 2026
8a47786
docs(proposals): add agent judge skill proposal
roark47 Jul 7, 2026
3cc2256
docs(proposals): address agent judge skill review
roark47 Jul 7, 2026
a7a9196
feat(judge): support judge-specific skills
roark47 Jul 7, 2026
96a3c98
feat(report): surface judge skill metadata
roark47 Jul 7, 2026
ed2466e
docs: document judge-specific skills
roark47 Jul 7, 2026
f16255c
chore(cli): annotate judge debug JSON serialization
roark47 Jul 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -63,3 +63,6 @@ e2e/coverage-report.json
node_modules/
docs/.vitepress/cache/
docs/.vitepress/dist/

# Local verification project
/verify-multi-turn/
16 changes: 15 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,21 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [0.4.0] - 2026-07-07

### Added
- **Multi-turn conversation evaluation** (`input.turns`): define sequential
user turns with per-turn `post_condition` gates (`must_contain_all`,
`must_contain_any`, `must_not_contain`, `on_fail: fail|skip_remaining`) and
`capture` for regex-based variable extraction between turns.
- Per-turn judge assertions: `turn_response_contains`,
`turn_response_not_contains`, `tool_called_in_turn`,
`tool_not_called_in_turn` — all scoped by turn number.
- Session resumption support for `claude_code` (`--resume`), `qodercli`
(`-r <session-id>`), and `codex` (`codex resume <thread-id>`) engines;
unsupported engines fall back to batch mode.
- `TurnResults` field in JSON, HTML, and JUnit reports (`turn_results` in
`result.json`; turn-scoped failure details in JUnit XML).

## [0.3.0] - 2026-07-03

Expand Down
8 changes: 8 additions & 0 deletions docs/design/custom-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,14 @@ The `local` transport runs a command via `runtime.Exec`. The command runs inside
current runtime, so it can access the runtime workspace, installed skills, fixtures,
MCP config, and environment variables.

Top-level `skills` are installed before the main run agent executes. If
`judge.type: agent_judge` declares `judge.skills`, those Skills are installed
separately before the judge agent runs. Both paths use the Agent adapter's
`InstallSkill` implementation; skill-up does not concatenate Skill files into
the prompt as a fallback. Custom and remote engines that need judge Skills must
make their agent process discover the installed Skill directory according to
their own conventions.

Example:

```yaml
Expand Down
Loading
Loading