This repository was archived by the owner on Aug 12, 2026. It is now read-only.
fix(verifier): tolerate incidental session/tracking writes in expect_only_v2 diff (ENVT-139728) - #137
Draft
hamid-netizen wants to merge 1 commit into
Draft
Conversation
…only_v2 diff Fixes ENVT-139728 Generated multi-app verifiers end with seed.diff(current, ignore_config, True).expect_only_v2(expected_changes), an allow-list-or-fail check that rejects ANY DB change not enumerated in expected_changes or covered by IgnoreConfig. The generated IgnoreConfigs omit routine session/tracking/login/audit tables, so a faithfully-correct agent performing the task produces incidental UI-interaction writes (login_activities, search_history, audit_log, navigation_items, ...) that fall outside both buckets and fail the whole verifier with 'Unexpected database changes detected'. Add a curated DEFAULT_INCIDENTAL_TABLES set to IgnoreConfig and treat incidental writes to those tables as ignored only when the table is not explicitly listed in allowed_changes. This preserves per-row validation of tables a verifier deliberately asserts on (including incidental ones) and still flags unexpected writes to non-incidental business tables. Pass IgnoreConfig(incidental_tables=set()) to opt out. Applied to the shared IgnoreConfig plus the empty-changes, unmentioned- tables row-count, and full-diff fallback paths in the standalone (fleet/verifiers/db.py), sync (fleet/resources/sqlite.py), and async (fleet/_async/resources/sqlite.py) SnapshotDiff implementations. Validation: 6 new focused tests in tests/test_expect_only.py cover incidental writes ignored, empty-changes tolerance, opt-out strictness, mentioned-incidental-table per-row validation, and business-write detection. Full expect_only/expect_exactly suites pass (142 passed, 11 skipped), no regressions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes ENVT-139728
Summary
Generated multi-app verifiers end with
seed.diff(current, ignore_config, True).expect_only_v2(expected_changes)— anallow-list-or-fail check that rejects ANY DB change not enumerated in
expected_changesor covered byIgnoreConfig. The generatedIgnoreConfigsomit routine session/tracking/login/audit tables, so a faithfully-correct agent
performing the task produces incidental UI-interaction writes
(
login_activities,search_history,audit_log,navigation_items, …) thatfall outside both buckets and fail the whole verifier with
"Unexpected database changes detected".This is the root cause of canonical bug ENVT-139728 (psi-consumer-finance
multi-app verifier) and the same-run sibling class (Harbor IgnoreConfig gaps,
over-strict diff validator across multi-app checks).
Fix
Add a curated
DEFAULT_INCIDENTAL_TABLESset toIgnoreConfigand treatincidental writes to those tables as ignored only when the table is not
explicitly listed in
allowed_changes:get full per-row validation.
IgnoreConfig(incidental_tables=set())opts out and restores strict behavior.Applied to the shared
IgnoreConfigplus the empty-changes, unmentioned-tablesrow-count, and full-diff fallback paths in the standalone
(
fleet/verifiers/db.py), sync (fleet/resources/sqlite.py), and async(
fleet/_async/resources/sqlite.py)SnapshotDiffimplementations — coveringboth the local targeted path and the production HTTP structured-diff path that
raised the reported error.
Validation (focused verifier tests / evidence-path replay)
6 new tests in
tests/test_expect_only.pyreplay the evidence path:test_expect_only_v2_ignores_incidental_session_writes— business writesenumerated + incidental login/session writes present → passes (the
reproduced failure case).
test_expect_only_v2_empty_changes_ignores_incidental_writes— Harbor-styleempty
expected_changeswith only incidental writes → passes.test_expect_only_v2_empty_changes_still_flags_business_writes— emptyexpected_changeswith a business write → fails (regression guard).test_expect_only_v2_incidental_opt_out_is_strict—IgnoreConfig(incidental_tables=set())restores pre-fix strict behavior →fails (polarity: opt-out == old behavior).
test_expect_only_v2_incidental_table_mentioned_still_validated— listing anincidental table in
allowed_changeswith a wrong field → fails (per-rowvalidation preserved).
test_expect_only_v2_still_flags_unexpected_business_writes— unexpectedinsert in a non-incidental table → fails.
Results:
6 passed(new); fulltest_expect_only.py+test_expect_exactly.py142 passed, 11 skipped, no regressions. No new rufffindings (14 pre-existing, unchanged).
Notes
verifier_versionsDB table (not in a repo); the gen prompt lives inBraintrust. The only repo-level runtime code that owns the reproduced failure
is this shared diff library, so the root-cause fix belongs here.
changes no environment repository and no
envs/<env_key>/path.🤖 Created via sandbox session