Skip to content
This repository was archived by the owner on Aug 12, 2026. It is now read-only.

fix(verifier): tolerate incidental session/tracking writes in expect_only_v2 diff (ENVT-139728) - #137

Draft
hamid-netizen wants to merge 1 commit into
mainfrom
bug-remediation/envt-139728-verifier-ignoreconfig
Draft

fix(verifier): tolerate incidental session/tracking writes in expect_only_v2 diff (ENVT-139728)#137
hamid-netizen wants to merge 1 commit into
mainfrom
bug-remediation/envt-139728-verifier-ignoreconfig

Conversation

@hamid-netizen

Copy link
Copy Markdown

Closes ENVT-139728

Summary

Generated multi-app verifiers end with
seed.diff(current, ignore_config, True).expect_only_v2(expected_changes) — an
allow-list-or-fail check that rejects ANY DB change not enumerated in
expected_changes or covered by IgnoreConfig. The generated IgnoreConfigs
omit routine session/tracking/login/audit tables, so a faithfully-correct agent
performing the task produces incidental UI-interaction writes
(login_activities, search_history, audit_log, navigation_items, …) that
fall outside both buckets and fail the whole verifier with
"Unexpected database changes detected".

This is the root cause of canonical bug ENVT-139728 (psi-consumer-finance
multi-app verifier) and the same-run sibling class (Harbor IgnoreConfig gaps,
over-strict diff validator across multi-app checks).

Fix

Add a curated DEFAULT_INCIDENTAL_TABLES set to IgnoreConfig and treat
incidental writes to those tables as ignored only when the table is not
explicitly listed in allowed_changes
:

  • Tables a verifier deliberately asserts on (including incidental ones) still
    get full per-row validation.
  • Unexpected writes to non-incidental business tables are still flagged.
  • IgnoreConfig(incidental_tables=set()) opts out and restores strict behavior.

Applied to the shared IgnoreConfig plus the empty-changes, unmentioned-tables
row-count, and full-diff fallback paths in the standalone
(fleet/verifiers/db.py), sync (fleet/resources/sqlite.py), and async
(fleet/_async/resources/sqlite.py) SnapshotDiff implementations — covering
both the local targeted path and the production HTTP structured-diff path that
raised the reported error.

Validation (focused verifier tests / evidence-path replay)

6 new tests in tests/test_expect_only.py replay the evidence path:

  1. test_expect_only_v2_ignores_incidental_session_writes — business writes
    enumerated + incidental login/session writes present → passes (the
    reproduced failure case).
  2. test_expect_only_v2_empty_changes_ignores_incidental_writes — Harbor-style
    empty expected_changes with only incidental writes → passes.
  3. test_expect_only_v2_empty_changes_still_flags_business_writes — empty
    expected_changes with a business write → fails (regression guard).
  4. test_expect_only_v2_incidental_opt_out_is_strict
    IgnoreConfig(incidental_tables=set()) restores pre-fix strict behavior →
    fails (polarity: opt-out == old behavior).
  5. test_expect_only_v2_incidental_table_mentioned_still_validated — listing an
    incidental table in allowed_changes with a wrong field → fails (per-row
    validation preserved).
  6. test_expect_only_v2_still_flags_unexpected_business_writes — unexpected
    insert in a non-incidental table → fails.

Results: 6 passed (new); full test_expect_only.py + test_expect_exactly.py

  • verifier suites → 142 passed, 11 skipped, no regressions. No new ruff
    findings (14 pre-existing, unchanged).

Notes

  • The verifier source for this task is LLM-generated and stored in the
    verifier_versions DB table (not in a repo); the gen prompt lives in
    Braintrust. The only repo-level runtime code that owns the reproduced failure
    is this shared diff library, so the root-cause fix belongs here.
  • The fixed-bug Playwright regression-artifact contract does not apply: this PR
    changes no environment repository and no envs/<env_key>/ path.

🤖 Created via sandbox session

…only_v2 diff

Fixes ENVT-139728

Generated multi-app verifiers end with
seed.diff(current, ignore_config, True).expect_only_v2(expected_changes),
an allow-list-or-fail check that rejects ANY DB change not enumerated in
expected_changes or covered by IgnoreConfig. The generated IgnoreConfigs
omit routine session/tracking/login/audit tables, so a faithfully-correct
agent performing the task produces incidental UI-interaction writes
(login_activities, search_history, audit_log, navigation_items, ...) that
fall outside both buckets and fail the whole verifier with
'Unexpected database changes detected'.

Add a curated DEFAULT_INCIDENTAL_TABLES set to IgnoreConfig and treat
incidental writes to those tables as ignored only when the table is not
explicitly listed in allowed_changes. This preserves per-row validation
of tables a verifier deliberately asserts on (including incidental ones)
and still flags unexpected writes to non-incidental business tables.
Pass IgnoreConfig(incidental_tables=set()) to opt out.

Applied to the shared IgnoreConfig plus the empty-changes, unmentioned-
tables row-count, and full-diff fallback paths in the standalone
(fleet/verifiers/db.py), sync (fleet/resources/sqlite.py), and async
(fleet/_async/resources/sqlite.py) SnapshotDiff implementations.

Validation: 6 new focused tests in tests/test_expect_only.py cover
incidental writes ignored, empty-changes tolerance, opt-out strictness,
mentioned-incidental-table per-row validation, and business-write
detection. Full expect_only/expect_exactly suites pass (142 passed,
11 skipped), no regressions.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant