release: 1.7.12

CHANGELOG.md

@@ -4,6 +4,12 @@ All notable changes to Lakebench are documented here.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [1.7.12] - 2026-10-23
+
+### Added
+
+- **`platform.compute.spark.executor_scratch` and the per-stage `bronze_executor_scratch`, `silver_executor_scratch` and `gold_executor_scratch` set each executor's scratch PVC (whole gibibytes, for example `200Gi`).** Every Spark job's scratch size was fixed by its built-in profile, with no key to raise it, so a job that outgrew its PVC could only fail: the AML scorer's 50Gi filled at scale 20 ("No space left on device" in the coverage aggregation's spill) and the run went unscored. A stage key sizes that stage's job in the run's mode; `executor_scratch` sizes every other job, including the scorer and time-travel reads. A set size is exact (not scaled by cores, scale or executor count), is counted by the capacity check and the scratch preflight, and is recorded in `config_snapshot.scratch.size_per_job` and the perf fingerprint, so runs with different scratch sizes do not compare as the same configuration. Setting one turns `platform.storage.scratch.enabled` on when it is unset; with it set to `false` the config is refused, since no executor would get a PVC. A config that sets none of these keys requests the same PVCs as before.
+
 ## [1.7.11] - 2026-10-23
 
 ### Breaking

docs/component-spark.md

@@ -84,13 +84,15 @@ Spark settings live under `platform.compute.spark`, `platform.storage.scratch`,
 | `bronze_executors`, `silver_executors`, `gold_executors` | `null` | Executor count of the stage's job in the run's mode (bronze-verify or bronze-ingest, silver-build or silver-stream, gold-finalize or gold-refresh). Set, it bypasses the scaling formula and, in continuous mode, the shared-cluster Lakebench cap. |
 | `bronze_executor_cores`, `silver_executor_cores`, `gold_executor_cores` | `null` | Continuous mode only: executor size, 1-16 cores. Memory and scratch per core follow the profile; the count is unchanged, so total cores grow. Batch jobs ignore it. |
 | `bronze_executor_memory`, `silver_executor_memory`, `gold_executor_memory` | `null` | Continuous mode only: heap per executor (e.g. `96g`), applied after the cores. The overhead keeps the profile's ratio to the heap; the capacity check and the node fit count the larger pod. Batch jobs ignore it. |
-| `platform.storage.scratch.enabled` | `false` | Portworx scratch PVCs for executors. Unset, a batch run at scale 50 and above turns it on. |
+| `executor_scratch` | `null` | Scratch PVC per executor for every Spark job, in whole gibibytes (e.g. `200Gi`). Replaces the profile's size outright, in either mode. |
+| `bronze_executor_scratch`, `silver_executor_scratch`, `gold_executor_scratch` | `null` | Scratch PVC per executor for the stage's job in the run's mode; wins over `executor_scratch` for that stage. |
+| `platform.storage.scratch.enabled` | `false` | Portworx scratch PVCs for executors. Unset, a batch run at scale 50 and above, or a config that sets a scratch size, turns it on. Set to `false` with a scratch size, the config is refused. |
 | `platform.storage.scratch.storage_class` | `px-csi-scratch` | Scratch StorageClass. Must be `repl=1`. |
 | `spark.conf` | `{}` | Spark conf merged over the job defaults (below). |
 
 - Executor count overrides above 28 are refused by the commands that change data; `destroy` and read commands drop them with a note.
 - Unset `<stage>_executor_cores`: bronze-ingest and silver-stream grow to 8 or 16 cores when the offered load needs more executors than the cap (keeping total cores), unless the stage's `<stage>_executors` count is set. gold-refresh is never grown.
-- Batch jobs take only a count override. The removed `platform.compute.spark.driver` and `.executor` blocks and `scratch.size` are listed under 1.7.0 in the [changelog](../CHANGELOG.md#170---2026-10-06).
+- Batch jobs take a count and a scratch size override. The removed `platform.compute.spark.driver` and `.executor` blocks and `scratch.size` are listed under 1.7.0 in the [changelog](../CHANGELOG.md#170---2026-10-06).
 
 ### Spark conf (S3A, shuffle, memory)
 
@@ -196,7 +198,7 @@ When enabled, each executor gets a dynamically provisioned PVC at `/tmp/spark-lo
 
 - **Batch, per-scale profile:** the stage's `scratch_gib_per_scale` x scale / executors, rounded up, between 50Gi and the profile's `scratch_size` (the ceiling). Examples: Customer 360 silver-build gets 8 x 50Gi at scale 1, 8 x 75Gi at scale 10 and 18 x 300Gi at scale 100; AML bronze-verify with 16 executors at scale 10 gets 50Gi each.
 - **No per-scale need** (c360 bronze-verify, 50Gi, and the streaming jobs): `scratch_size`. A widened streaming executor's PVC grows with its cores.
-- To change scratch, change the executor count (`platform.compute.spark.<stage>_executors`). There is no size field.
+- **Set in the config:** `platform.compute.spark.<stage>_executor_scratch`, else `executor_scratch`, replaces the size above for every executor of the job. It is exact: not scaled by cores, scale or executor count. `executor_scratch` is the only control for jobs outside the three stages, such as the AML scorer.
 - `metrics.json` records the size per job as `config_snapshot.scratch.size_per_job`.
 - The StorageClass must use `repl=1`; replication only doubles storage for recomputable shuffle data.
 - `lakebench deploy` only checks that the StorageClass exists. A cluster admin creates it once with `lakebench admin install --component scratch-storage-class`.

docs/configuration.md

@@ -87,7 +87,7 @@ The removed keys, with what to do instead (generated from each model's
 | `platform.compute.spark.silver_stream_executors` | use silver_executors, which sizes silver-stream in continuous mode and silver-build in batch mode. |
 | `platform.storage.s3.secret_ref` | lakebench never reads an existing Secret: deploy writes the S3 Secret from access_key and secret_key. Set those instead (use ${VAR} substitution, for example access_key: "${S3_ACCESS_KEY}", to keep the keys out of the file). |
 | `platform.storage.scratch.create_storage_class` | lakebench creates no StorageClass; a cluster admin runs 'lakebench admin install --component scratch-storage-class' once. |
-| `platform.storage.scratch.size` | per-job scratch comes from the job profiles (silver-build: 60 GiB x scale / executors, 50-300Gi). |
+| `platform.storage.scratch.size` | each executor's scratch PVC is platform.compute.spark.executor_scratch (or a stage's *_executor_scratch), else its job profile's size. |
 | `secret_ref` | lakebench never reads an existing Secret: deploy writes the S3 Secret from access_key and secret_key. Set those instead. |
 | `version` | there is one config schema and nothing read the number; delete the key. |
 | `workload.customer360.channels` | the customer360 generator never read it; delete the key. |
@@ -287,7 +287,7 @@ platform:
 
     scratch:
       # enabled: true                 # Portworx scratch PVCs; unset, it turns on for a batch run at scale 50 and above
-      storage_class: px-csi-scratch   # PVC size per executor: the job profile's
+      storage_class: px-csi-scratch   # PVC size per executor: compute.spark.*executor_scratch, else the job profile's
 
   compute:
     spark:
@@ -301,8 +301,7 @@ platform:
 
       # Per-stage executor counts (null = auto). One key per stage; the run's
       # mode picks the job: bronze-verify or bronze-ingest, silver-build or
-      # silver-stream, gold-finalize or gold-refresh. Per-executor memory and
-      # PVC are fixed from proven profiles.
+      # silver-stream, gold-finalize or gold-refresh.
       bronze_executors: null
       silver_executors: null
       gold_executors: null
@@ -313,6 +312,12 @@ platform:
       silver_executor_cores: null
       gold_executor_cores: null
 
+      # Scratch PVC per executor (null = the job profile's size). Exact, in
+      # either mode; a stage key wins for its stage. Setting one turns
+      # scratch on when platform.storage.scratch.enabled is unset.
+      executor_scratch: null          # e.g. 200Gi
+      silver_executor_scratch: null
+
       # Global driver overrides (null = profile default).
       driver_memory: null
       driver_cores: null
@@ -588,6 +593,10 @@ Scratch PVCs for Spark shuffle data. Only needed with Portworx or similar CSI.
 | `platform.compute.spark.bronze_executor_memory` | string or null | `null` | advanced | Continuous mode: override bronze-ingest heap per executor (a Spark size, for example `32g`); batch mode ignores it. The memory overhead keeps the profile's ratio to the heap, and the capacity check and the node fit count the larger pod. Null = the profile's per-core share of the executor size. |
 | `platform.compute.spark.silver_executor_memory` | string or null | `null` | advanced | Continuous mode: override silver-stream heap per executor (a Spark size, for example `96g`); batch mode ignores it. The memory overhead keeps the profile's ratio to the heap, and the capacity check and the node fit count the larger pod. For a financial run that leaves `silver_max_rows_per_trigger` unset, a larger heap raises the derived cap up to its measured bound. Null = the profile's per-core share of the executor size. |
 | `platform.compute.spark.gold_executor_memory` | string or null | `null` | advanced | Continuous mode: override gold-refresh heap per executor (a Spark size, for example `96g`); batch mode ignores it. The memory overhead keeps the profile's ratio to the heap, and the capacity check and the node fit count the larger pod. Null = the profile's per-core share of the executor size. |
+| `platform.compute.spark.executor_scratch` | string or null | `null` | advanced | Scratch PVC per executor for every Spark job (whole gibibytes, for example `200Gi`). A stage's `*_executor_scratch` wins for that stage's jobs. Setting any scratch size turns `platform.storage.scratch.enabled` on when it is unset; with it set to `false` the config is refused. Null = each job profile's size. |
+| `platform.compute.spark.bronze_executor_scratch` | string or null | `null` | advanced | Scratch PVC per executor for the bronze stage (for example `200Gi`): bronze-verify in batch mode, bronze-ingest in continuous mode. Null = `executor_scratch`, else the profile's size. |
+| `platform.compute.spark.silver_executor_scratch` | string or null | `null` | advanced | Scratch PVC per executor for the silver stage (for example `300Gi`): silver-build in batch mode, silver-stream in continuous mode. Null = `executor_scratch`, else the profile's size. |
+| `platform.compute.spark.gold_executor_scratch` | string or null | `null` | advanced | Scratch PVC per executor for the gold stage (for example `200Gi`): gold-finalize in batch mode, gold-refresh in continuous mode. Null = `executor_scratch`, else the profile's size. |
 | `platform.compute.spark.driver_memory` | string or null | `null` | advanced | Global driver memory override (e.g., `16g`). Null = profile default. |
 | `platform.compute.spark.driver_cores` | integer or null | `null` | advanced | Override driver cores (1--16). Null = profile default (typically 4). |
 

docs/troubleshooting.md

@@ -485,8 +485,11 @@ platform:
       storage_class: px-csi-scratch
 ```
 
-The per-executor scratch size comes from the job profile and should not be
-reduced; see [Job Profiles](component-spark.md#job-profiles). Silver-build
+The per-executor scratch size comes from the job profile unless
+`platform.compute.spark.executor_scratch` or a stage's `*_executor_scratch`
+sets it; do not set it below the profile's. To fix a "No space left on device"
+in one job, raise that stage's key, or `executor_scratch` for a job outside
+the three stages. See [Job Profiles](component-spark.md#job-profiles). Silver-build
 asks for 60 GiB x scale / executors per executor, between 50Gi and 300Gi.
 Raising `platform.compute.spark.silver_executors` makes each PVC smaller but
 does not shrink the total (scale 100: 28 x 215Gi = 6,020 Gi, against 18 x

src/lakebench/__init__.py

@@ -1,3 +1,3 @@
 """Lakebench - CLI tool for deploying and benchmarking lakehouse architectures on Kubernetes."""
 
-__version__ = "1.7.11"
+__version__ = "1.7.12"

src/lakebench/cli/_init.py

@@ -217,6 +217,7 @@ def first_day_dict(
         "platform.compute.spark.bronze_executor_memory",
         "platform.compute.spark.silver_executor_memory",
         "platform.compute.spark.gold_executor_memory",
+        "platform.compute.spark.executor_scratch",
         "platform.compute.spark.driver_memory",
         "platform.compute.spark.driver_cores",
         "platform.compute.postgres.storage",

src/lakebench/config/autosizer.py

@@ -426,10 +426,11 @@ def resolve_auto_sizing(
     # -- Schema-specific overrides --
     # Workload schemas that differ from Customer360 on baseline resource shape
     # override defaults here. Financial raises the Spark Thrift memory.
-    # Per-executor scratch PVCs are job.py:scratch_executor_size (the stage's
-    # scratch_gib_per_scale x scale over its executors, up to the profile's
-    # scratch_size) with _SCHEMA_PROFILE_OVERRIDES on top, applied at
-    # manifest-build time in modules/pipeline_engines/spark/job.py.
+    # Per-executor scratch PVCs are job.py:executor_scratch_size: the size the
+    # config sets, else the stage's scratch_gib_per_scale x scale over its
+    # executors, up to the profile's scratch_size, with
+    # _SCHEMA_PROFILE_OVERRIDES on top, applied at manifest-build time in
+    # modules/pipeline_engines/spark/job.py.
     # -- Spark Thrift on Delta --
     # Runs before the schema overrides so the financial 24g heap still wins.
     delta_change = _apply_delta_thrift_default(config, cluster_capacity)
@@ -587,12 +588,20 @@ def _apply_delta_thrift_default(
 def _autoscratch_applies(config: LakebenchConfig) -> bool:
     from lakebench.config.schema import PipelineMode
 
-    return (
+    return _scratch_size_set(config) or (
         config.architecture.pipeline.mode == PipelineMode.BATCH
         and config.architecture.workload.datagen.scale >= _AUTOSCRATCH_MIN_SCALE
     )
 
 
+def _scratch_size_set(config: LakebenchConfig) -> bool:
+    """Whether the config sets a ``platform.compute.spark.*executor_scratch``."""
+    from lakebench.config.schema import SCRATCH_OVERRIDE_KEYS
+
+    spark = config.platform.compute.spark
+    return any(getattr(spark, key) is not None for key in SCRATCH_OVERRIDE_KEYS)
+
+
 def scratch_will_be_enabled(config: LakebenchConfig) -> bool:
     """Whether the run uses scratch PVCs, before or after auto-sizing."""
     scratch = config.platform.storage.scratch
@@ -602,7 +611,8 @@ def scratch_will_be_enabled(config: LakebenchConfig) -> bool:
 
 
 def _apply_scratch_autoenable(config: LakebenchConfig) -> str | None:
-    """Enable scratch PVCs on a batch run at scale 50 or above.
+    """Enable scratch PVCs on a batch run at scale 50 or above, or when the
+    config sets an executor scratch size.
 
     Only sets ``platform.storage.scratch.enabled`` when the user did not set
     it. ``storage_class`` and ``provisioner`` keep their defaults. ``plan``,
@@ -616,6 +626,11 @@ def _apply_scratch_autoenable(config: LakebenchConfig) -> str | None:
     scratch = config.platform.storage.scratch
     if not _set_if_default(scratch, "enabled", True):
         return None
+    if _scratch_size_set(config):
+        return (
+            "platform.storage.scratch.enabled=True (the config sets an executor "
+            "scratch size, which needs a scratch PVC per executor)"
+        )
     return (
         f"platform.storage.scratch.enabled=True (scale {scale:g}: "
         "silver-build shuffle exceeds pod ephemeral storage without a "