Python: [Feature]: Document a third-party Python evaluator that preserves evidence metadata

[roli-lpci]

Context: Agent Framework 1.17.0 and Hermes Rubric 1.2.1 on Python 3.10. Filing this before code or a PR, per the contribution guidance.

Description

Agent Framework already defines an experimental, provider-neutral Python Evaluator protocol and structured EvalResults, EvalItemResult, and EvalScoreResult types. I could not find a third-party example showing how to preserve additional provenance—evidence citations, assessed coverage, and a run receipt—without reducing the result to an unexplained number.

Proposed scope

A thin optional-dependency example or integration implementing the current protocol exactly:

async def evaluate(
    self,
    items: Sequence[EvalItem],
    *,
    eval_name: str,
) -> EvalResults:
    ...

It would invoke Hermes Rubric through assess_async(..., rubric=caller_rubric, backend=caller_backend), then map:

• aggregate score to EvalScoreResult.score;
• per-dimension scores to RubricScore;
• evidence citations, coverage, limitations, and receipt to EvalScoreResult.sample or EvalItemResult.metadata, as maintainers prefer;
• Hermes AssessmentError to item status="error" with error_code and error_message, with no numeric fallback.

It would provide no default rubric, backend, threshold, or pass predicate. Hermes FeedbackPolicy, if supplied, would only generate deterministic post-assessment feedback; it would not be treated as a pass/fail policy.

Decision needed

EvalScoreResult.passed may be None, while documented EvalItemResult.status values are pass, fail, or error. Before implementation, should this adapter:

1. require a caller-supplied pass predicate so item status is defined, or
2. remain an external example until maintainers specify the supported non-gating mapping?

If wanted, should it live under python/samples or remain an external package referencing the protocol?

Tests

• pinned imports with Agent Framework 1.17.0 and Hermes Rubric 1.2.1 on Python 3.10;
• evidence, dimensions, coverage, and receipt preserved;
• partial coverage remains explicit;
• malformed/provider failures become explicit errors without numeric fallback;
• missing optional dependency fails clearly.

Limitations

This is one evaluator option, not a Foundry replacement or parity claim. Results depend on the caller’s rubric and backend and remain advisory. Python only; no adoption, certification, correctness, or safety claim.

I will not open a PR until a maintainer confirms the route and the required item-status mapping.