Repository navigation
fix(evaluator): do not pass correct on an assertion the run contradicts - #310
Open
rahulver551 wants to merge 1 commit into
Open
rahulver551 wants to merge 1 commit into
rahulver551 wants to merge 1 commit into
Conversation
…icts
A seller that quoted 3999 cents for the 3990-cent task, and a buyer that
asserted `correct: true` without comparing, produced a run in which every
stage passed, the verdict was `passed`, and `nandatown verify` reported
that the evaluator reproduces the recorded result. The bundle contradicts
itself: the buyer's terminal acknowledgement records
`{"correct": true, "total_cents": 3999, "expected_total_cents": 3990}`
and the seller's processed acknowledgement records `total_cents: 3999`.
Under 0.5.0 `correct` is decided by the buyer's boolean alone. Reproduced
through the real runner with a fixture seller and a fixture buyer.
Under 0.6.0 ("consistent_verdict") a `correct: true` passes only when
nothing the run recorded contradicts it:
- the buyer's own terminal acknowledgement recording `total_cents`, or
`expected_total_cents`, other than the task's expected total fails
`correct`: that record is the buyer's, and it sits beside the assertion;
- a seller's processed acknowledgement recording another `total_cents`
leaves `correct` inconclusive: the town does not record the response's
total, so the two participants' records disagree and neither decides.
Totals are read only as JSON integers, the way counts already are; a
string, float, boolean or container states nothing and changes nothing.
A `correct: false` is judged as before. 0.2.0 to 0.5.0 bundles replay on
the assertion alone, unchanged; an existing 0.5.0 bundle of the
reproduction still verifies.
Tests: tests/test_correct_contradicted.py (16), observed failing on main
first (4 failed), including one through `run_town` with a new
`wrongtotal` mode on tests/fixtures/track_seller_variant.py and a new
tests/fixtures/credulous_buyer.py. Full suite: 1768 passed, 2 skipped;
the two read-only-home tests in test_url_credentials.py fail only when run
as root and pass as an ordinary user, as on main.
Not included: SHIPPED_EVALUATOR_VERSIONS in bundle.py, whose comment says
to add a version when it merges.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…icts
A seller that quoted 3999 cents for the 3990-cent task, and a buyer that asserted
correct: truewithout comparing, produced a run in which every stage passed, the verdict waspassed, andnandatown verifyreported that the evaluator reproduces the recorded result. The bundle contradicts itself: the buyer's terminal acknowledgement records{"correct": true, "total_cents": 3999, "expected_total_cents": 3990}and the seller's processed acknowledgement recordstotal_cents: 3999. Under 0.5.0correctis decided by the buyer's boolean alone. Reproduced through the real runner with a fixture seller and a fixture buyer.Under 0.6.0 ("consistent_verdict") a
correct: truepasses only when nothing the run recorded contradicts it:total_cents, orexpected_total_cents, other than the task's expected total failscorrect: that record is the buyer's, and it sits beside the assertion;total_centsleavescorrectinconclusive: the town does not record the response's total, so the two participants' records disagree and neither decides.Totals are read only as JSON integers, the way counts already are; a string, float, boolean or container states nothing and changes nothing. A
correct: falseis judged as before. 0.2.0 to 0.5.0 bundles replay on the assertion alone, unchanged; an existing 0.5.0 bundle of the reproduction still verifies.Tests: tests/test_correct_contradicted.py (16), observed failing on main first (4 failed), including one through
run_townwith a newwrongtotalmode on tests/fixtures/track_seller_variant.py and a new tests/fixtures/credulous_buyer.py. Full suite: 1768 passed, 2 skipped; the two read-only-home tests in test_url_credentials.py fail only when run as root and pass as an ordinary user, as on main.Not included: SHIPPED_EVALUATOR_VERSIONS in bundle.py, whose comment says to add a version when it merges.