Typed criterion verdicts on grading results

[solvemproblr]

Problem

Grading results lost most of their structure on the way to the trace:

• LLMJudgeGrader's per-criterion verdicts lived in SubScore.metadata (excluded from serialization), so the trace viewer showed rubric results as an untyped JSON blob at best.
• combine_any / combine_all collapsed their inputs into a single subscore, stashing input names and metadata in a metadata dict and flattening criteria into one list — the grading history was unrecoverable.
• Because SubScore.metadata didn't serialize, combine() copied it into EvaluationResult.info keyed by subscore name — a second untyped side-channel.

Solution

Make SubScore a tree node and type the rubric verdicts:

• CriterionResult — one checked requirement with passed, weight, reason. Lives on SubScore.criteria, so verdicts ride with the grader that produced them.
• SubScore.children + SubScore.aggregation — combine_any / combine_all now return a node with aggregation: "any" | "all" and their inputs preserved verbatim as children (names deduped per sibling group). Combinators nest, so arbitrary grading trees serialize whole. A validator rejects aggregation without children and warns when the value disagrees with max/min of the children.
• SubScore.metadata serializes — the EvaluationResult.info copy and the combinator metadata blobs are gone; each node carries its own details.
• Grader.compute_score returns float | SubScore — replaces the (float, metadata_dict) tuple and the magic metadata["criteria"] key. grade() stamps name / weight / _parameters onto the result; LLMJudgeGrader and BashGrader build their SubScore directly.

Outcome

EvaluationResult.subscores is now a recursive structure carrying every grader run, its criteria, its metadata, and how values were combined.

---

Note

Medium Risk
Public grader API changes (tuple returns and EvaluationResult.info metadata side-channel removed) may break custom graders or trace consumers; scope is evaluation/serialization only, not auth or data paths.

Overview
Grading outputs are now a serializable tree instead of flattened metadata blobs, so traces can show rubric verdicts and how subscores were combined.

CriterionResult is exported and attached on SubScore.criteria. LLMJudgeGrader emits one per rubric line (with passed, weight, reason) instead of stuffing verdicts into excluded metadata.

SubScore gains children and aggregation ("any" / "all"). combine_any / combine_all keep their inputs as child nodes (with deduped names) rather than collapsing into metadata dicts. A validator requires children when aggregation is set and warns if value disagrees with min/max of children.

combine no longer copies per-subscore metadata into EvaluationResult.info; metadata stays on each subscore and serializes with the breakdown (previously metadata was excluded from dumps).

Grader.compute_score now returns float | SubScore (the (float, metadata_dict) tuple is removed). grade() wraps floats or stamps name / weight / _parameters onto a returned SubScore. BashGrader follows the same SubScore-first shape.

Docs in graders.mdx document CriterionResult and the tree fields.

^{Reviewed by Cursor Bugbot for commit 165a89d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
hud	🟢 Ready	View Preview	Jul 9, 2026, 11:40 PM