Repository navigation
release: 1.7.11 - #54
Closed
AndrewSillifant wants to merge 10 commits into
Closed
AndrewSillifant wants to merge 10 commits into
AndrewSillifant wants to merge 10 commits into
Conversation
kubectl logs -f is severed at the kubelet streamingConnectionIdleTimeout (4h on the test cluster) and nothing restarted it, so a 24h record held only the first ~4h of batches. A kept-up bronze then read as a fraction of itself: a live AML run reported ingest_ratio 0.24 while its lag never exceeded ~250s and silver never ran dry. - _driver_log_capture: restart a severed tail in place with --since-time from the last stamp already captured, so the file stays one continuous log per pod. A still-live tail is left alone; a new pod uid is still a resubmit with its own file. Add _last_stamp and tests for it and the existing (untested) merge_logs seam dedup. - collector: record max_batch_id per stage and refuse to score ingest_ratio when the bronze log has a gap over 10% of its batches (bronze_log_incomplete), reporting ingest_ratio_unscored_reason instead of a false shortfall verdict. verdict surfaces it as a warning, not a FAIL. Also add the 1.7.10 silver.accounts balance-state columns (last_entry_seq, last_book_ts) to the ensure_column upgrade path in both the batch and stream financial scripts, so a catalog created before 1.7.10 and reused across the upgrade does not fail the first write / micro-batch on a schema mismatch.
A continuous run's corpus grows for the whole window, so a full-scan or running-balance query eventually exceeds any fixed time limit. The gate turned that into FAILED-and-unscored, discarding the run's silver, freshness and detection evidence: the live 1.7.9/1.7.10 24h runs failed for this reason while bronze and silver kept up. Two further faults fed it: the continuous in-stream rounds fell through to run_power's 300 s default while financial batch used 900 s, and QpH was computed over the queries that passed, so a round that lost its heavy queries kept a higher QpH and a degrading series could read as improving. Separate the query SLA from the pipeline verdict: - QueryExecutorResult/QueryResult carry timed_out; the four executor timeout paths set it (Trino client timeout AND the server-side EXCEEDED_TIME_LIMIT that query_max_run_time raises ~5 s earlier, spark-thrift, duckdb remote + local). - is_sla_crossing() is the single definition of a crossing: a FAILED query that timed out AND passed in some round of this run. Both the CLI gate (_benchmark_gate_problems, passed_ever) and the metrics.json verdict re-derivation (_failed_queries) call it, so the exit code and the stored/report verdict cannot disagree. A timeout on a query that never passed, and every non-timeout error, still fail (the 8d3d383 1-of-8 hole stays closed). Batch mode passes passed_ever=None: a timeout fails as before. - round_is_scorable() excludes any round with a real failure (not a Q9 contention miss, not a demoted crossing) from the composite/median QpH, trend and aggregate, so a degrading series cannot read as improving. - bench_query_timeout(cfg) (900 financial / 300 c360) is now the one limit both batch and continuous use; continuous records it in benchmark_query_timeout_seconds. It is also the limit a crossing is measured against. - Each crossing is recorded on its round (query_sla_crossings) with that round's silver_data_file_count. Also drop test_delta_thrift_q2_failure_fails_the_run: the gate has no tolerated-query carve-out left, so it only re-asserted the generic any-failure path.
Version bump, CHANGELOG, and the AML correctness-contract rows for the 1.7.11 changes: the query time-limit crossing (a crossing is recorded, not a failure; the verdict rests on the pipeline gates), the shared 900 s / 300 s continuous benchmark timeout, the ingest_ratio not-scored-on-incomplete-log guard, and the silver.accounts upgrade-path fix.
The pre-compaction warm-up and round still inlined the 900/300 financial ternary; route them through bench_query_timeout(cfg) so the one-source-of- truth the 1.7.11 change establishes covers every batch benchmark path. Identical value, no behavioural change.
Line-wrapping from ruff format, and three mypy fixes: carry max_batch_id on StreamingJobMetrics (not only StageMetrics) so the parse path types, rename the resumed-tail local so it does not narrow _start_tail's Popen | None return, and build passed_ever as set[str].
…gest gap Both are in the ingest_ratio path. 1. Bronze resume-seam rows were double-counted. committed_writes returned every write event, and the callers sum rows per event, so the ~1 s --since-time overlap a resumed capture re-fetches counted the seam batch's rows twice. committed_writes now returns one write per batch id (silver already dedups the same way via a per-ident dict), so a duplicate from a resume seam, or any cause, is counted once. 2. The gap guard false-tripped on sparse bronze batch ids. Bronze empty and replay-skipped batches log an id but no "writing" line, so the counted ids are not contiguous even on a complete log; (max_id + 1 - count) / (max_id + 1) over 0.1 then suppressed ingest_ratio on a run with many empty triggers. The guard now reads the largest run of consecutive missing ids (max_batch_id_gap, computed at parse time): a severed capture drops thousands in one run, scattered empties make runs of 1, so one distinguishes a truncated log from a sparse one. Bound is 50 consecutive ids. Tests: bronze seam duplicate counts each batch once; scattered empty batches (gap of 1) do not false-trip; a 240-id gap does.
…tch_id The CLI gate, the verdict re-derivation and the QpH median each built the set of queries that passed in some round, with slightly different None handling, and the QpH path rebuilt it once per round. rounds_passed_ever is now the one builder, called once per evaluation. max_batch_id was recorded per stage but read by nothing; the gap guard uses max_batch_id_gap. CHANGELOG: describe the shipped gap rule (a run of 50 consecutive missing batch ids) and the per-batch-id bronze count. correctness.md: state the crossing rule in one line.
User docs narrated version history ("since 1.7.1", "before 1.7", the
recorded 1.6 figures and their caveat repeated across the AML pages),
bug archaeology, design decisions and internal identifiers. An evaluator
needs what the workload does and what each number means. Remove the
history, narration and identifiers; keep every limit and cost statement.
- AML.md is an index: overview (why) -> AML.md (map) -> aml/*.md (detail).
docs/README marks each overview as the place to start.
- seed-policy.md is cut to what an evaluator needs; the held-out seed
protocol moves to development.md.
- aml-overview: drop the claim that the bronze-retention threshold is part
of what a comparison reveals (it is a fixed value).
- scoring.md: finish the sentence that ended mid-clause.
- Config field descriptions fixed at source in schema.py; configuration.md,
cli-reference.md and exit-codes.md regenerated (the last two were already
stale on main).
- gen_sizing_tables no longer targets README.md, which dropped its sizing
table in the docs rewrite; the unused minimums renderer goes with it.
The suite pinned many behaviours several times, tested libraries and its own tooling, and asserted wording and mock plumbing no user sees. Cut those; keep every scoring, gate, destroy-safety, ownership, exit-code and held-out-seed test. - Developer tooling: drop the pre-push hook, gitleaks history and baseline, and calendar-leakage tests; trim release_gate, package_guard (held-out seed checks kept) and check_version. Drop the fixture scrubber's rule tests; keep fixture-is-clean and the golden records. - Review-session files (saf*, cfg*, rpt1_er*, *_v16, lb148, evd5, category1, equivalence_review_fixes) are merged into their module's test file or renamed by topic, without ticket ids. - Same-branch parametrization, tautologies against a module's own constants, pydantic behaviour, exact console/HTML wording, call counts and private helpers: shrunk or cut. The continuous sizing grid and the interrupt and scale-invariance harness runs take far fewer cases. - Dead code the cut tests held up: config/_readers.py (only the schema walk imported it), collector.SNAPSHOT_SOURCE_FIELDS, and the LB_SILVER_TEST_ALLOW_EMPTY bypass in common.py, which nothing set. Unit suite: 4726 -> 4081 tests.
- The reference-gate monthly-unit and seed tests run counts-only where the scored statistics are not what they check; one scored run remains as the write-path proof, plus the seed-claim run that alone proves corpus_seed_verified. - Single-test AML files that each started a child JVM are merged into the file that already exercises the same function (gold finalize and refresh version filters, current_balance, profile replay, the two Delta stream race tests, refuse-fresh-checkpoint). - Rebuilds that only proved a comparator can see a change, the 'ordered' incremental-detection layout that 'restart' covers, a duplicate delta epoch rebuild, path-cap and fingerprint tests pinned elsewhere, and asserts on Spark's xxhash64 and Iceberg's overwrite semantics: cut. - Customer-scope schema and reason-code checks read the module fixture instead of re-running every rule. About 25 minutes less per pyspark version.
Collaborator
Author
|
Superseded: reopening with the stale stream-offset Spark test fixed so the tier is fully green. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
1.7.11 makes the continuous AML verdict measure the pipeline instead of two artefacts, and trims the docs and tests.
Measurement fixes (Breaking, see CHANGELOG)
query_sla_crossings; a query that never passed, or any non-timeout error, still fails. Oneis_sla_crossingdefinition is shared by the CLI gate, the stored verdict andlakebench report, so they cannot disagree. Rounds that lost a query are left out of the QpH median and trend.ingest_ratiois not scored on an incomplete bronze log.kubectl logs -fis cut at the kubelet idle timeout (4 h); the capture now resumes in place, bronze rows are counted once per batch id, and a run of 50+ consecutive missing batch ids marks the ratio not-scored instead of reporting a false shortfall (a 24 h run read 0.24 while bronze kept up).Fixed
silver.accountscreated before 1.7.10 getslast_entry_seq/last_book_tson startup (batch and stream).Docs
User docs state current behaviour: version history, decision narration and internal identifiers removed; AML.md is an index under the overview; the held-out seed protocol moves to development.md; a misleading retention claim and a truncated sentence fixed.
cli-reference.mdandexit-codes.mdregenerated (stale on main); the sizing generator no longer targets README.Tests
Unit suite 4726 -> 4081, Spark tier 387 -> 365 and about 10 minutes faster per pyspark version. Cut: tests of developer tooling, pydantic and the fixture scrubber; same-branch parametrization; wording, call-count and private-helper asserts; repeated Spark proofs and child JVMs. Review-session-named test files merged into module files or renamed by topic. Dead code they held up removed (
config/_readers.py, an unused snapshot constant, an unset silver test bypass). Every scoring, gate, destroy-safety, ownership, exit-code and held-out-seed test is kept.Verification
test_expiry_floored_at_live_stream_offset, which fails on main and was the one failure when 1.7.10 shipped; this branch does not touch it or maintenance code.