Add new datasets to benchmark. Add Kaggle support

.github/workflows/lib-lint-and-test.yaml

@@ -202,6 +202,13 @@ jobs:
       - check-paths
     if: needs.check-paths.outputs.run_benchmark_dataset_scripts == 'true'
     timeout-minutes: 30
+    # Kaggle API token for the (optional, credential-gated) real-download
+    # test in test_prepare_brain_tumor_instseg.py. Only set for same-repo runs —
+    # GitHub never exposes secrets to `pull_request` workflows triggered from
+    # forks, so that one test skips itself gracefully (via a `skipif` check)
+    # rather than failing on external contributors' PRs.
+    env:
+      KAGGLE_API_TOKEN: ${{ secrets.KAGGLE_API_TOKEN }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

library/Justfile

@@ -18,7 +18,7 @@ _ensure_uv:
 [arg("device", short="d", long="device")]
 venv device="cpu": _ensure_uv
     uv lock --check
-    uv sync --frozen --extra {{ device }} --extra ultralytics
+    uv sync --frozen --extra {{ device }} --extra ultralytics --extra benchmark
 
 # Generate or update the uv lock file
 venv-lock: _ensure_uv

library/benchmark_catalog.yaml

@@ -10,9 +10,26 @@
 #
 # Fields:
 #   name        – unique identifier (referenced from benchmark_manifest.yaml)
-#   script      – path to preparation script (relative to library/ root)
+#   script      – path to preparation script (relative to library/ root).
+#                 Mutually exclusive with `local_path` — exactly one must be set.
+#   local_path  – path to an already-prepared dataset directory, used as-is
+#                 with no script execution. Supports ${VAR}/~ expansion so the
+#                 same entry resolves differently per machine/CI runner (e.g.
+#                 "${GETITUNE_BENCHMARK_EXTERNAL_DATA}/my_dataset"). Use this
+#                 for datasets that were prepared out-of-band (manually, or on
+#                 a shared/network mount) and should never be re-downloaded.
+#   raw_dir     – optional, only valid together with `script`. Path (also
+#                 supports ${VAR}/~ expansion) to a pre-fetched raw archive or
+#                 directory, forwarded to the script as `--raw-dir` so it can
+#                 skip its own network download step (e.g. for credentialed
+#                 sources like Kaggle) while still running the same
+#                 transform/export logic. If the path doesn't exist, the
+#                 script falls back to its normal download behavior.
 #   size_tier   – tiny | small | medium | large
 #   url         – upstream source the preparation script downloads from
+#
+# See src/getitune/benchmark/README.md ("Datasets requiring credentials or
+# manual placement") for a walkthrough of `local_path` / `raw_dir` usage.
 
 version: 1
 
@@ -37,6 +54,48 @@ datasets:
     compatible_tasks:
       - classification/multi_class_cls
 
+  - name: paddy_disease
+    script: "scripts/benchmark_datasets/prepare_paddy_disease.py"
+    size_tier: small
+    url: "https://huggingface.co/datasets/anthony2261/paddy-disease-classification"
+    description: "Paddy Disease Classification — public HF mirror of a rice disease dataset with image-level class labels."
+    compatible_tasks:
+      - classification/multi_class_cls
+
+  - name: chest_ct
+    script: "scripts/benchmark_datasets/prepare_chest_ct.py"
+    size_tier: medium
+    url: "https://huggingface.co/datasets/Mahadih534/Chest_CT-Scan_images-Dataset"
+    description: "Chest CT Scan Images — public HF mirror of a 4-class chest CT classification dataset."
+    compatible_tasks:
+      - classification/multi_class_cls
+
+  - name: aircraft
+    script: "scripts/benchmark_datasets/prepare_aircraft.py"
+    raw_dir: "${GETITUNE_BENCHMARK_EXTERNAL_DATA}/aircraft_raw"
+    size_tier: large
+    url: "https://www.kaggle.com/datasets/seryouxblaster764/fgvc-aircraft"
+    description: >-
+      FGVC-Aircraft — 10,200 aircraft images with official train/val/test splits and
+      hierarchical labels (variant / family / manufacturer). This benchmark uses the
+      variant level and requires a Kaggle account for automatic download, or a
+      pre-extracted raw directory at $GETITUNE_BENCHMARK_EXTERNAL_DATA/aircraft_raw.
+    compatible_tasks:
+      - classification/multi_class_cls
+
+  - name: food41
+    script: "scripts/benchmark_datasets/prepare_food41.py"
+    raw_dir: "${GETITUNE_BENCHMARK_EXTERNAL_DATA}/food41_raw"
+    size_tier: large
+    url: "https://www.kaggle.com/datasets/kmader/food41"
+    description: >-
+      Food-101 — 101 food categories with official train/test splits, sourced from
+      a Kaggle mirror. This benchmark uses a stratified validation split from the
+      training set and supports Kaggle auto-download or a pre-extracted raw
+      directory at $GETITUNE_BENCHMARK_EXTERNAL_DATA/food41_raw.
+    compatible_tasks:
+      - classification/multi_class_cls
+
   - name: oxford_pets_hlabel
     script: "scripts/benchmark_datasets/prepare_oxford_pets_hlabel.py"
     size_tier: large
@@ -94,3 +153,79 @@ datasets:
     description: "Vehicles (Roboflow-100) — 4,058 road-scene images with bounding-box annotations across 12 vehicle types (cars, buses, trucks). Large multi-class on-road detection benchmark."
     compatible_tasks:
       - detection
+
+  - name: pothole
+    script: "scripts/benchmark_datasets/prepare_pothole.py"
+    size_tier: tiny
+    url: "https://huggingface.co/datasets/keremberke/pothole-segmentation"
+    description: "Pothole Segmentation — 90 road-surface images with polygon annotations for potholes. Tiny real-world segmentation benchmark."
+    compatible_tasks:
+      - instance_segmentation
+      - semantic_segmentation
+      - detection # polygons -> bounding boxes
+
+  - name: pcb_defect
+    script: "scripts/benchmark_datasets/prepare_pcb_defect.py"
+    size_tier: small
+    url: "https://huggingface.co/datasets/keremberke/pcb-defect-segmentation"
+    description: "PCB Defect Segmentation — 189 close-up PCB images with polygon annotations across 4 defect classes. Small industrial segmentation benchmark."
+    compatible_tasks:
+      - instance_segmentation
+      - semantic_segmentation
+      - detection # polygons -> bounding boxes
+
+  - name: satellite_building
+    script: "scripts/benchmark_datasets/prepare_satellite_building.py"
+    size_tier: large
+    url: "https://huggingface.co/datasets/keremberke/satellite-building-segmentation"
+    description: "Satellite Building Segmentation — 9,665 aerial images with polygon annotations for building footprints. Medium-scale segmentation benchmark."
+    compatible_tasks:
+      - instance_segmentation
+      - semantic_segmentation
+      - detection # polygons -> bounding boxes
+
+  - name: corrosion
+    script: "scripts/benchmark_datasets/prepare_corrosion.py"
+    size_tier: tiny
+    url: "https://huggingface.co/datasets/rkumari/corrosion_segment"
+    description: "Corrosion segmentation — 44 images with per-pixel severity masks across 4 classes. Tiny semantic-segmentation benchmark."
+    compatible_tasks:
+      - semantic_segmentation
+
+  - name: crack_semantic
+    script: "scripts/benchmark_datasets/prepare_crack_semantic.py"
+    size_tier: large
+    url: "https://huggingface.co/datasets/varcoder/crack-segmentation-dataset"
+    description: "Crack segmentation — 11,298 images with binary crack masks. Large semantic-segmentation benchmark."
+    compatible_tasks:
+      - semantic_segmentation
+
+  - name: brain_tumor_instseg
+    script: "scripts/benchmark_datasets/prepare_brain_tumor_instseg.py"
+    raw_dir: "${GETITUNE_BENCHMARK_EXTERNAL_DATA}/brain_tumor_instseg_raw"
+    size_tier: large
+    url: "https://www.kaggle.com/datasets/pkdarabi/medical-image-dataset-brain-tumor-segmentation"
+    description: >-
+      Brain Tumor instance segmentation — ~1,744 MRI images with polygon annotations
+      (3 tumor-likelihood classes) from a Kaggle-gated Roboflow YOLO export. Requires a
+      Kaggle account: either configure Kaggle API credentials (see the preparation
+      script's docstring) for automatic download, or place a pre-fetched copy at
+      $GETITUNE_BENCHMARK_EXTERNAL_DATA/brain_tumor_instseg_raw (raw_dir is forwarded to
+      the script and used instead of downloading when present).
+    compatible_tasks:
+      - instance_segmentation
+
+  - name: cardd
+    script: "scripts/benchmark_datasets/prepare_cardd_instseg.py"
+    raw_dir: "${GETITUNE_BENCHMARK_EXTERNAL_DATA}/cardd_instseg_raw"
+    size_tier: large
+    url: "https://www.kaggle.com/datasets/issamjebnouni/cardd"
+    description: >-
+      CarDD instance segmentation — 4,000 images with COCO polygon annotations across 6
+      damage classes (dent, scratch, crack, glass shatter, lamp broken, tire flat), from
+      a Kaggle-gated COCO-format mirror. Requires a Kaggle account: either configure
+      Kaggle API credentials (see the preparation script's docstring) for automatic
+      download, or place a pre-fetched copy at $GETITUNE_BENCHMARK_EXTERNAL_DATA/cardd_instseg_raw
+      (raw_dir is forwarded to the script and used instead of downloading when present).
+    compatible_tasks:
+      - instance_segmentation

library/benchmark_manifest.yaml

@@ -33,7 +33,10 @@ experiments:
         priority: extended
         recipe: classification/multi_class_cls/efficientnet_b0.yaml
     datasets:
-      - flowers102
+      - paddy_disease
+      - chest_ct
+      - aircraft
+      - food41
     criteria:
       accuracy_metric: accuracy
       thresholds:
@@ -76,6 +79,9 @@ experiments:
         "torch:test/latency": { compare: "<=", margin: 0.15 }
   detection:
     models:
+      - name: rfdetr_nano
+        priority: core
+        recipe: detection/rfdetr_nano.yaml
       - name: yolox_s
         priority: core
         recipe: detection/yolox_s.yaml
@@ -139,10 +145,32 @@ experiments:
       - name: yolo26_m
         priority: extended
         recipe: detection/yolo26_m.yaml
+      - name: yolo11_n
+        priority: extended
+        recipe: detection/yolo11_n.yaml
+      - name: yolo11_s
+        priority: extended
+        recipe: detection/yolo11_s.yaml
+      - name: yolo11_m
+        priority: extended
+        recipe: detection/yolo11_m.yaml
+      - name: yolo11_l
+        priority: extended
+        recipe: detection/yolo11_l.yaml
+      - name: yolo11_x
+        priority: extended
+        recipe: detection/yolo11_x.yaml
+      - name: rfdetr_2xlarge
+        priority: extended
+        recipe: detection/rfdetr_2xlarge.yaml
     datasets:
+      - axial_mri
+      - wgisd
       - bccd
+      - cable_damage
+      - vehicles
     criteria:
-      accuracy_metric: map_50
+      accuracy_metric: map
       thresholds:
         "training:val/{metric}": { compare: ">=", margin: 0.10 }
         "torch:test/{metric}": { compare: ">=", margin: 0.10 }
@@ -175,7 +203,10 @@ experiments:
         priority: extended
         recipe: semantic_segmentation/litehrnet_x.yaml
     datasets:
-      - oxford_pets
+      - corrosion
+      - pcb_defect
+      - crack_semantic
+      - satellite_building
     criteria:
       accuracy_metric: Dice
       thresholds:
@@ -187,6 +218,9 @@ experiments:
         "torch:test/latency": { compare: "<=", margin: 0.15 }
   instance_segmentation:
     models:
+      - name: rfdetr_seg_nano
+        priority: core
+        recipe: instance_segmentation/rfdetr_seg_nano.yaml
       - name: rfdetr_seg_small
         priority: core
         recipe: instance_segmentation/rfdetr_seg_small.yaml
@@ -220,10 +254,30 @@ experiments:
       - name: yolo26_m_seg
         priority: extended
         recipe: instance_segmentation/yolo26_m_seg.yaml
+      - name: yolo11_n_seg
+        priority: extended
+        recipe: instance_segmentation/yolo11_n_seg.yaml
+      - name: yolo11_s_seg
+        priority: extended
+        recipe: instance_segmentation/yolo11_s_seg.yaml
+      - name: yolo11_m_seg
+        priority: extended
+        recipe: instance_segmentation/yolo11_m_seg.yaml
+      - name: yolo11_l_seg
+        priority: extended
+        recipe: instance_segmentation/yolo11_l_seg.yaml
+      - name: yolo11_x_seg
+        priority: extended
+        recipe: instance_segmentation/yolo11_x_seg.yaml
     datasets:
       - wgisd
+      - pothole
+      - pcb_defect
+      - brain_tumor_instseg
+      - cardd
+
     criteria:
-      accuracy_metric: map_50
+      accuracy_metric: map
       thresholds:
         "training:val/{metric}": { compare: ">=", margin: 0.10 }
         "torch:test/{metric}": { compare: ">=", margin: 0.10 }

library/docs/design/automated-model-benchmarking.md

@@ -338,6 +338,54 @@ def provision_datasets(
     ...
 ```
 
+Provisioning is resilient per-dataset: if one entry fails (missing credentials, a
+transient network error, an unresolvable `local_path`/`raw_dir`), the error is logged and
+that entry is simply omitted from the returned mapping rather than aborting the whole
+batch. The runner already treats a missing dataset name as "skip the experiments that
+need it" (§8), so a single problematic dataset never blocks unrelated experiments.
+
+#### 5.3.2 Credentialed, Manually-Placed, and Local Datasets
+
+Not every dataset can be freely auto-downloaded: some require an account/license
+(e.g. Kaggle), and some are large enough that re-fetching them on every run is wasteful.
+Rather than build a generic pluggable "dataset source" abstraction, the catalog schema
+gets two additional, optional fields that compose with the existing script contract:
+
+- **`local_path`** (alternative to `script`) — a directory that is already fully
+  prepared (placed manually, mounted from a shared/network location, or copied from a
+  prior script run). No script runs; the path is used as-is, after `${VAR}`/`~`
+  expansion so the same entry resolves differently per machine/CI runner. Mutually
+  exclusive with `script`.
+- **`raw_dir`** (optional, alongside `script`) — a pre-fetched raw archive/directory,
+  forwarded to the script as `--raw-dir`. This lets a script skip only its network fetch
+  step (e.g. because the raw data was obtained once, manually, with a person's own
+  Kaggle credentials) while still running its transform/export logic, keeping that logic
+  shared, version-controlled, and testable. It's a best-effort accelerant: if it doesn't
+  resolve, the script falls back to its normal download behavior instead of failing.
+
+```yaml
+datasets:
+  - name: brain_tumor_instseg
+    script: "scripts/benchmark_datasets/prepare_brain_tumor_instseg.py"
+    raw_dir: "${GETITUNE_BENCHMARK_EXTERNAL_DATA}/brain_tumor_instseg_raw"
+    size_tier: small
+```
+
+Scripts sourcing from Kaggle use `getitune.benchmark.dataset_helpers.download_kaggle_dataset()`,
+which checks for `kagglehub` and API credentials (`KAGGLE_API_TOKEN` or
+`~/.kaggle/access_token`) up front and raises a clear, actionable error rather than a
+bare dependency failure. `kagglehub` itself is an opt-in optional dependency extra
+(`benchmark`; e.g. `uv sync --extra benchmark`), not installed by default. In CI, a `KAGGLE_API_TOKEN`
+repository secret is wired into the `benchmark-dataset-scripts` job; the corresponding
+real-download test is additionally `skipif`-gated on credentials being present, so
+forked-repo PRs (which never receive secrets) skip it instead of failing.
+
+This directly answers §17 Open Question #1 for the credentialed/oversized subset of
+datasets: rather than depending on a single hosting service, `local_path` lets any
+externally-managed location (shared drive, network mount, manually-populated directory)
+serve as a dataset's source of truth, while `raw_dir` keeps the credentialed/huge case
+inside the same reviewable script pipeline as every other dataset.
+
 ---
 
 ## 6. Benchmark Manifest
@@ -1770,7 +1818,7 @@ results/
 
 | #   | Question                                            | Proposed Default                                                        | Notes                                                                                                                                                                                                                                                                                                                                                                                             |
 | --- | --------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| 1   | Where to host benchmark datasets?                   | Produced by in-repo preparation scripts (`scripts/benchmark_datasets/`) | Scripts download from any source, transform, and place data locally. No dependency on a single hosting service.                                                                                                                                                                                                                                                                                   |
+| 1   | Where to host benchmark datasets?                   | Produced by in-repo preparation scripts (`scripts/benchmark_datasets/`) | Scripts download from any source, transform, and place data locally. No dependency on a single hosting service. Credentialed (e.g. Kaggle) or oversized datasets additionally support `local_path` (fully externally-managed, no script) and `raw_dir` (skip only the script's network fetch) — see §5.3.2.                                                                                    |
 | 2   | Remote MLflow server or local-only?                 | Start local (`mlruns/`), move to remote server in Phase 5               | Avoids infra dependency in early phases.                                                                                                                                                                                                                                                                                                                                                          |
 | 3   | How many seeds per experiment?                      | 3                                                                       | Balances statistical confidence vs. CI cost. 5 seeds would be better but almost doubles runtime.                                                                                                                                                                                                                                                                                                  |
 | 4   | Should the benchmark block PR merges on regression? | No (advisory only)                                                      | Flaky GPU tests + margin of error make hard-gating risky initially. Revisit after baseline stability is established.                                                                                                                                                                                                                                                                              |

library/pyproject.toml

@@ -127,6 +127,10 @@ ci_publish = [
     "build==1.2.2.post1",
 ]
 
+benchmark = [
+    "kagglehub>=0.4.1",
+]
+
 [project.scripts]
 getitune = "getitune.cli:main"
 otx = "getitune.cli:main"