release: 1.7.11

CHANGELOG.md

@@ -4,6 +4,18 @@ All notable changes to Lakebench are documented here.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [1.7.11] - 2026-10-22
+
+### Breaking
+
+- **A continuous benchmark query that exceeds its time limit after passing in an earlier round is recorded as a measured SLA crossing, not a run failure, and the pipeline verdict now rests on the pipeline gates alone.** A continuous run's corpus grows for the whole window, so a full-scan or running-balance query (`FQ1`, `FQ3`, `FQ4`, `FQ7`) eventually crosses any fixed per-query limit; the benchmark gate turned the first such crossing into a FAILED, unscored run, discarding the run's silver, freshness and detection evidence. A failed query is now a crossing, exempt from the gate, only when it failed by exceeding its time limit **and** succeeded in some round of this run; the round and that round's silver data-file count are recorded under the round's `query_sla_crossings`. A timeout on a query that never passed, and every non-timeout error, still fail the run, so a run where queries genuinely broke is still marked FAILED (the 1-of-8-passing hole that the gate was added to close stays closed). A round that lost a query to a crossing is left out of the composite/median QpH, the QpH trend and the aggregate, because QpH is computed over the queries that passed and would otherwise read a degrading series as improving. The CLI exit, the stored verdict and `lakebench report` share one crossing definition, so they cannot disagree. Batch mode is unchanged: with no earlier round to cross from, a timeout fails as before. A run whose every benchmark query stays inside its limit for the whole window scores exactly as before.
+- **The continuous in-stream benchmark rounds now use the same per-query time limit as the batch benchmark (900 s for financial, 300 s for Customer 360), and record it in `benchmark_query_timeout_seconds`.** The in-stream rounds passed no limit and fell through to the 300 s default, so a financial continuous run held its analytical queries to 300 s while a financial batch run of the same queries allowed 900 s; a query between those bounds timed out in continuous but not in batch, which read as a failed query rather than a slow one. The limit is now one value for both modes and is also the limit a crossing is measured against. A continuous Customer 360 run is unchanged (its limit was already 300 s); a financial continuous run's query timings are now comparable to the batch run's.
+- **`ingest_ratio` is reported not-scored when the bronze driver log is incomplete, instead of as a shortfall computed from a truncated row count.** `kubectl logs -f` is severed at the kubelet's streaming-connection idle timeout (four hours on the validated cluster) and nothing restarted it, so a run longer than that held only the batches before the cut; the ratio's numerator then counted a fraction of the rows bronze actually ingested and a bronze that kept up read as having fallen behind (a 24-hour AML run reported `ingest_ratio` 0.24 while its datagen-to-bronze lag never exceeded about 250 s and silver never ran dry). The run now restarts a severed capture in place from the last timestamp it holds, so the log stays continuous, and the collector refuses to score `ingest_ratio` when the bronze log still has a run of 50 or more consecutive batch ids missing, recording `bronze_log_incomplete` and `ingest_ratio_unscored_reason` rather than a false verdict. Scattered single missing ids (empty or replay-skipped bronze batches) do not trip it. Bronze rows are now counted once per batch id, so a line repeated in the driver log is not counted twice. A run whose bronze log is complete (any run under four hours, or any run once a severed capture resumes) is scored as before.
+
+### Fixed
+
+- **A `silver.accounts` table created before 1.7.10 and reused across the upgrade no longer fails the first batch build or stream micro-batch on a schema mismatch.** 1.7.10 added `last_entry_seq` and `last_book_ts` to `silver.accounts`, but neither the batch build nor the stream added them to the catalog's existing table on the upgrade path, so a reused pre-1.7.10 `silver.accounts` left `CREATE TABLE IF NOT EXISTS` a no-op and the first accounts overwrite (batch) or prior-state read (stream) failed on the missing columns. Both scripts now add the two columns on startup, the same idempotent upgrade step already used for the other post-DDL columns. A fresh deployment, or one created on 1.7.10 or later, is unaffected.
+
 ## [1.7.10] - 2026-10-20
 
 ### Breaking

docs/README.md

@@ -12,12 +12,15 @@ workload experiments.
 
 ## Workloads
 
-- [Customer 360 overview](benchmarks/c360-overview.md): what the workload is,
-  what each stage stresses, and why you would run it
+Start with a workload's overview page, then follow its links into the
+specification.
+
+- [Customer 360 overview](benchmarks/c360-overview.md) (start here): what the
+  workload is, what each stage stresses, and why you would run it
 - [Customer 360](benchmarks/C360.md): data model, pipeline stages, queries,
   metrics and comparability
-- [AML overview](benchmarks/aml-overview.md): what the workload is, what each
-  stage stresses, and why you would run it
+- [AML overview](benchmarks/aml-overview.md) (start here): what the workload
+  is, what each stage stresses, and why you would run it
 - [AML](benchmarks/AML.md): data model, seed policy, detection rules,
   scoring and comparability
 - [AML Scoring](aml-scoring.md): how to read recall and precision, what

docs/aml-scoring.md

@@ -8,7 +8,7 @@ The workload reads pacs.008 wire messages into bronze, builds silver entity, acc
 
 **Recall and precision are uncalibrated and in-sample.** No held-out [Level-2](glossary.md#level-2) result is published. The default seed, 43, is the calibration corpus: the generator, the rule thresholds and the reference features were developed on it. A seed-43 figure says what the pipeline does on that corpus, not how the rules generalise.
 
-The scorer is the Spark job [`score_financial.py`](../src/lakebench/spark/scripts/score_financial.py). A batch `lakebench run` submits it after gold-finalize. It joins `gold.alerts` to the generator's manifest on payment UETRs, with no time window.
+The scorer is a Spark job that a batch `lakebench run` submits after gold-finalize. It joins `gold.alerts` to the generator's manifest on payment UETRs, with no time window.
 
 | Scorecard figure | `recall.json` key | What it is |
 |---|---|---|
@@ -20,20 +20,15 @@ The scorer is the Spark job [`score_financial.py`](../src/lakebench/spark/script
 
 None of these is a production false-positive rate (next section). Each key is defined in [AML spec 8.4](benchmarks/aml/scoring.md#84-aml-scoring-reported-a-batch-run-without-a-result-fails).
 
-Two figures that look like results are not:
-
-- **Seven-day-window recall.** `aml_queries.py` holds Trino SQL templates with a 7-day window that only unit tests run. A recall figure with a 7-day window did not come from Lakebench's scorer.
-- **Pattern span** (`pattern_span_s`). It describes the planted data, not the stack, and is not time to detect. Detection latency is `time_to_detect_seconds` (continuous only). See [AML spec 6](benchmarks/aml/queries.md#6-query-set).
-
-The rules, their target typologies and what each detects are in [AML spec 4.2](benchmarks/aml/rules.md#42-detection-rules). The planted typologies, including the sanctions and PEP screening track, are in [sections 3.6 and 3.8](benchmarks/aml/typologies.md#36-typologies). A typology no rule targets is listed in `UNMAPPED_TYPOLOGIES` with a reason; its recall is untestable, not zero.
+The rules, their target typologies and what each detects are in [AML spec 4.2](benchmarks/aml/rules.md#42-detection-rules). The planted typologies, including the sanctions and PEP screening track, are in [sections 3.6 and 3.8](benchmarks/aml/typologies.md#36-typologies). A typology no rule targets has untestable recall, not zero.
 
 ## Why benchmark precision is not the FP rate ops teams care about
 
 Precision here is the share of a rule's alerts that land on payments the manifest tags as planted, in a synthetic bronze layer. It does not compare with the 90%+ false-positive rate reported for production alert queues.
 
 - **One baseline distribution.** Non-planted payments come from one log-normal amount distribution with weekend and hour multipliers. Real customers vary by account type, industry, season and known-good vendors, with a long tail of unusual but legitimate activity that drives ops-queue false positives. The generator does not make that tail, so a rule at 0.95 precision here says nothing about how many legitimate customers it would alert on in production.
 - **No "legitimate but suspicious" label.** Every non-planted row is clean. In production many alerts hit customers a reviewer clears; benchmark precision has no counterpart for that bucket.
-- **The rules were written against this generator.** The thresholds in `detection_rules.py` were tuned for the planted typologies and claim nothing about the right tuning for a bank.
+- **The rules were written against this generator.** The rule thresholds were tuned for the planted typologies and claim nothing about the right tuning for a bank.
 
 Use precision to compare stacks and catch regressions between releases: a change between two configurations means the detector's separation from the same synthetic baseline changed. For production questions, labelled transaction data from the bank is needed.
 
@@ -42,7 +37,6 @@ Not measured:
 - **Real alert queues.** One baseline distribution and about a dozen planted shapes, against a bank's far wider transaction mix.
 - **Rule tuning quality.** Thresholds were chosen for reasonable recall on this generator.
 - **Rule logic against real typologies.** The rules are illustrative W-rule patterns for benchmarking pipeline throughput, not production detectors.
-- **Typologies missing from both `RULE_TARGETS` and `UNMAPPED_TYPOLOGIES`.** No test checks that every typology the manifest emits is in one of them.
 
 ## Held-out evaluation
 
@@ -52,12 +46,12 @@ No Level-2 result is published. The registered held-out evaluation and robustnes
 
 What protects the held-out corpora:
 
-- The repository stores only salted hashes of the evaluation and robustness seeds (`src/lakebench/spark/data/aml/heldout_hashes.json`), next to `aml_preregistration.json`, which fixes the gate constants and when each look may be taken.
+- The repository stores only salted hashes of the evaluation and robustness seeds.
 - Config load refuses a held-out seed unless `workload.datagen.corpus_role` declares the matching role.
 - With the role declared, `run`, `benchmark`, `query` and the `financial` subcommands still refuse the corpus with exit 2 before any cluster call. Only `lakebench generate --registered-corpus --yes` writes it, and only `scripts/aml_gate.py --registered` scores it.
 - The in-run scorer, bronze-verify and the reference job read every manifest row and refuse a corpus from a held-out or spent seed, whatever seed the config claims.
 
-The held-out protocol itself is maintained internally. [AML spec 3.3](benchmarks/aml/seed-policy.md#33-seed-policy) has every rule, refusal and exit code.
+[AML spec 3.3](benchmarks/aml/seed-policy.md#33-seed-policy) has the seed rules.
 
 ## See also
 

docs/benchmarking/continuous.md

@@ -94,6 +94,5 @@ With `max_files_per_trigger` set (the trickle) and the pipeline keeping pace, or
 - `limits.bound` line: `trickle: max_files_per_trigger N (auto), the pipeline kept pace`. Not in `limits.bound_kinds`, so not in the experiment identity.
 - Capped rows (`capped_by` names the trickle): `sustained_throughput_rps`, `pipeline_throughput_gb_per_second`, compute efficiency, `corpus_drain_seconds`.
 - The report labels those rows/s, GB/s and efficiency figures "BOUNDED BY: trickle (offered load, not capacity)", with the tooltip "trickle N files per trigger; this is the offered load, not infrastructure capacity".
-- Pre-1.7 records get this when read.
 
 AML continuous recall is scored over covered instances: planted cases whose every transaction and participant reached the scored [tick](../glossary.md#tick)'s tables. See [AML spec 8.4](../benchmarks/aml/scoring.md#84-aml-scoring-reported-a-batch-run-without-a-result-fails).

docs/benchmarking/query-benchmark.md

@@ -79,7 +79,6 @@ Five rounds or more for trend analysis: `run_duration >= warmup + 4 * (interval
 **Gold event age** (diagnostic, not freshness): query time minus `MAX(interaction_date)`, day resolution. It follows the corpus's event dates: a corpus dated 2024-12-18 to 2025-01-01 reads ~635 days in 2026.
 
 - Median: `pipeline_benchmark.diagnostics.query_time_event_age_seconds`. Per round: `round_meta.gold_event_age_seconds`.
-- Records before 1.6 wrote it as `query_time_freshness_seconds` / `gold_freshness_seconds` and printed it as the continuous Pipeline Score. The score line now shows `data_freshness_seconds`.
 
 ### In-stream QpH basis
 

docs/benchmarking/records.md

@@ -50,7 +50,7 @@ Each run saves `lakebench-output/runs/run-<id>/metrics.json`:
 
 ## Storage I/O
 
-The `io` block holds the storage traffic the engines reported for the run, derived from the rest of the record when it is written; records from before 1.7.2 lack it. Bytes are what the storage clients count; pod network bytes are not added.
+The `io` block holds the storage traffic the engines reported for the run, derived from the rest of the record when it is written. Bytes are what the storage clients count; pod network bytes are not added.
 
 - `intervals`: one per benchmark query (any instrumented engine), one for datagen, and one per 15 s step of each Spark stage that gained bytes. Each has `source`, `stage`, `label` (`benchmark`, `pre_compaction` or `round_<n>`), the `query`, `start` and `end` (UTC, from the engine's clock; null when not recorded), `read_bytes` and `write_bytes`, the basis of each, and `complete`.
 - `null` bytes mean not measured; `0` means measured and zero.

docs/benchmarking/scorecard.md

@@ -25,7 +25,6 @@ Typical time to value: 200-600 s at scale 10 (~100 GB), 1200-3600 s at scale 100
 - Times are mapped to the Lakebench host's clock through the API server's clock offset.
 - `timing_source`: `driver_container`, `spark_application`, or `poll` when no consistent cluster time could be read.
 - `timing_resolution_seconds`: 2 s from the cluster, else the poll interval.
-- Records before 1.6 ended every stage on the 15 s job-monitor poll: stage seconds and time to value read up to 15 s per stage late and include the gold driver-log fetch. Do not compare them with later records.
 
 ## Per-stage metrics
 

docs/benchmarking/verdict.md

@@ -60,6 +60,6 @@ If the corpus runs out after half the window, the run passes (`window_arrival_fr
 
 - Only an executor override that ran with fewer executors is a mismatch.
 - An unmet request is a qualifier (`verdict.qualifiers.requested_effective`) and a report warning, such as "gold_strategy: requested two_phase_agg, ran simple_agg (auto)".
-- Incremental gold a job chose automatically is labelled even under `auto` (it aggregates part of silver; current gold jobs no longer make that choice). Incremental gold for a multi-cycle cycle is by design and is not.
+- Incremental gold a job chose automatically is labelled even under `auto` (it aggregates part of silver). Incremental gold for a multi-cycle cycle is not labelled.
 - Mismatches are decided at read; `experiment.requested_effective_mismatches` keeps the run's list.
 - A mismatch never fails a run or enters the experiment identity. Other caps are in `experiment.limits` and `limits.bound`.

docs/benchmarks/AML.md

@@ -8,15 +8,10 @@
 New here? Start with the [AML overview](aml-overview.md), which explains what
 the workload is, what each stage stresses, and why you would run it.
 
-The specification for engineers who run or compare the benchmark and
-contributors who implement it on a new component. It describes the code of
-this release. Python paths are relative to `src/lakebench/`, generator paths
-to `datagen_rs/`. For how to read the scores, see
+The specification for engineers who run or compare the benchmark. It
+describes the code of this release. For how to read the scores, see
 [aml-scoring.md](../aml-scoring.md).
 
-Figures labelled "recorded" come from three 1.6 run records
-([recorded 1.6 figures](aml/limitations.md#recorded-16-figures)).
-
 ## Pages
 
 1. [Purpose and scope](#1-purpose-and-scope)
@@ -33,35 +28,17 @@ Figures labelled "recorded" come from three 1.6 run records
 
 ## 1. Purpose and scope
 
-The workload models a bank's transaction-monitoring (TM) back office over a
-five-year archive of ISO 20022 pacs.008 credit transfers. A synthetic
-generator plants known money-laundering typologies into a clean payment
-stream. The pipeline:
-
-- lands the messages (bronze);
-- normalises them into payments, parties, accounts, statements, a
-  counterparty graph and profiles (silver);
-- runs nine detection rules and a simulated alert-triage, case and SAR
-  workflow (gold).
-
-A fixed query set is then timed against the result.
-
 Per architecture (catalog x table format x pipeline engine x query engine, on
-a known system), it measures:
+a known system), the workload measures:
 
 | Mode | Measures |
 |---|---|
 | Batch | time to turn a fixed raw corpus into queryable gold (`time_to_value_seconds`); data volume and requested compute used; query-set speed (`composite_qph`) |
 | Continuous | gold staleness while datagen writes at the scale's offered load (`data_freshness_seconds`); time from a planted pattern's payments landing to its alert (`time_to_detect_seconds`); the ingest rate bronze holds; QpH while tables are written |
 
-- Figures are valid only when the verdict passes
-  ([5](aml/correctness.md#5-correctness-contract)).
-- A batch comparison also needs the same query set and alert set
-  ([10](aml/comparability.md#10-comparability)). Lakebench does not compare
-  query answers.
-- A continuous run's alerts depend on when detection and TM passes ran
-  relative to arrival, so a continuous comparison cannot establish matching
-  results.
+Figures are valid only when the verdict passes
+([5](aml/correctness.md#5-correctness-contract)); the conditions for comparing
+two runs are in [10](aml/comparability.md#10-comparability).
 
 What it does not claim:
 
@@ -93,8 +70,7 @@ What it does not claim:
 ## 11. Supported compositions
 
 AML runs on Iceberg only. Of the 11 structurally valid tuples, 8 are
-workload-compatible, in batch and continuous (`config/schema.py`,
-`WORKLOAD_TABLE_FORMATS` and `WORKLOAD_MODES`).
+workload-compatible, in batch and continuous.
 
 The generated [support table](../compatibility-matrix.md#support-states)
 lists support states per recipe and mode. AML is supported only on the cells
@@ -113,9 +89,9 @@ it lists; every other valid combination is unverified.
 | hive-delta-spark-{trino,thrift,none} | -- | -- | unsupported, refused at load: the AML stage scripts and DDL are Iceberg-only |
 
 - The pipeline engine is Spark in every recipe. Every recipe uses
-  `apache/spark:4.1.3-python3` (`config/recipes.py`); a config that sets
-  `images.spark` runs that image.
-- The release matrix's AML rows (`config/support.py`): batch
+  `apache/spark:4.1.3-python3`; a config that sets `images.spark` runs that
+  image.
+- The release matrix's AML rows: batch
   hive-iceberg-spark-trino at scale 1 and 10, batch
   polaris-iceberg-spark-trino, continuous hive-iceberg-spark-trino and
   continuous polaris-iceberg-spark-trino.