Add new datasets to benchmark. Add Kaggle support

[kprokofi]

Summary

• Update dataset list for detection, instance seg, semantic segmentation and classification tasks
• Add support Kaggle API
• Add deferred option to models to exclude them from rotation, but to be able to run them manually
• Add new data_group parameter to explicitly control weekly datasets list
• Update documentation

How to test

Checklist

• The PR title and description are clear and descriptive
• I have manually tested the changes
• All changes are covered by automated tests
• All related issues are linked to this PR (if applicable)
• Documentation has been updated (if applicable)

[github-actions]

⏱️ Backend import time — app.main

Accelerator	Cumulative (s)	Self (s)
cpu	4.608	0.003
xpu	4.502	0.003
cuda	5.345	0.003

[Copilot]

Pull request overview

This PR expands the getitune benchmarking system by adding multiple new benchmark datasets and introducing Kaggle-backed dataset provisioning support (via kagglehub), alongside catalog enhancements for externally managed datasets.

Changes:

• Extend benchmark catalog/provisioning to support local_path (pre-prepared datasets) and raw_dir (skip credentialed/slow downloads), with per-dataset resilient provisioning.
• Add several new dataset preparation scripts (HuggingFace + Kaggle sources) plus unit tests for the new provisioning paths and dataset converters.
• Update benchmark manifests/catalog, documentation, CI wiring, and dependency lock/config to include the new datasets and Kaggle support.

Reviewed changes

Copilot reviewed 35 out of 36 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
.github/workflows/lib-lint-and-test.yaml	Adds Kaggle token env + changes benchmark-dataset job install command.
library/Justfile	Adjusts uv sync extras used by just venv.
library/benchmark_catalog.yaml	Adds new datasets + documents local_path/raw_dir catalog fields.
library/benchmark_manifest.yaml	Updates benchmark experiment dataset/model lists (classification/detection/segmentation).
library/docs/design/automated-model-benchmarking.md	Documents resilient provisioning and credentialed/manual dataset options.
library/pyproject.toml	Adds benchmark optional extra with kagglehub.
library/scripts/benchmark_datasets/hf_mask_helper.py	New helper to build semantic-seg datasets from HF image+mask parquet mirrors.
library/scripts/benchmark_datasets/prepare_aircraft.py	New Kaggle-backed FGVC Aircraft preparation script.
library/scripts/benchmark_datasets/prepare_axial_mri.py	Updates HF parquet paths to include data/ prefix.
library/scripts/benchmark_datasets/prepare_brain_tumor_instseg.py	New Kaggle-backed Roboflow-style YOLO polygon instance-seg dataset script.
library/scripts/benchmark_datasets/prepare_cable_damage.py	Updates HF parquet paths to include data/ prefix.
library/scripts/benchmark_datasets/prepare_cardd.py	New Kaggle-backed YOLO detection dataset script.
library/scripts/benchmark_datasets/prepare_cardd_instseg.py	New Kaggle-backed COCO instance-seg dataset script.
library/scripts/benchmark_datasets/prepare_chest_ct.py	New zip-archive classification dataset preparation script.
library/scripts/benchmark_datasets/prepare_corrosion.py	New HF mask-pair semantic-seg dataset script.
library/scripts/benchmark_datasets/prepare_crack_semantic.py	New HF mask-pair semantic-seg dataset script (thresholded masks + val split).
library/scripts/benchmark_datasets/prepare_food41.py	New Kaggle-backed Food-101 dataset script with stratified val split.
library/scripts/benchmark_datasets/prepare_paddy_disease.py	New HF classification dataset script with stratified re-splitting.
library/scripts/benchmark_datasets/prepare_pcb_defect.py	New Roboflow HF polygon segmentation dataset script.
library/scripts/benchmark_datasets/prepare_pothole.py	New Roboflow HF polygon segmentation dataset script.
library/scripts/benchmark_datasets/prepare_satellite_building.py	New Roboflow HF polygon segmentation dataset script.
library/scripts/benchmark_datasets/prepare_vehicles.py	Updates HF parquet paths to include data/ prefix.
library/scripts/benchmark_datasets/roboflow_hf_helper.py	Generalizes helper to support repo-relative parquet paths + optional polygon segmentation.
library/src/getitune/backend/lightning/engine.py	Adjusts Lightning precision type import used under TYPE_CHECKING.
library/src/getitune/benchmark/README.md	Adds guidance for local_path/raw_dir and Kaggle dataset setup.
library/src/getitune/benchmark/catalog.py	Adds local_path/raw_dir, path-template expansion, raw-dir forwarding, and resilient batch provisioning.
library/src/getitune/benchmark/dataset_helpers.py	Adds --raw-dir, raw source resolution helper, and Kaggle download helper + credential check.
library/tests/unit/benchmark/test_catalog.py	Adds tests for new catalog validation, path expansion, local datasets, raw_dir forwarding, and provisioning resilience.
library/tests/unit/benchmark/test_dataset_helpers.py	Adds tests for --raw-dir, raw source resolution, Kaggle credential detection, and Kaggle download helper behavior.
library/tests/unit/scripts/test_prepare_aircraft.py	Adds unit tests for aircraft dataset script helpers and CLI failure mode.
library/tests/unit/scripts/test_prepare_brain_tumor_instseg.py	Adds unit tests for YOLO polygon parsing + raw-dir path + optional network test.
library/tests/unit/scripts/test_prepare_cardd.py	Adds unit tests for YOLO bbox parsing + raw-dir path + optional network test.
library/tests/unit/scripts/test_prepare_cardd_instseg.py	Adds unit tests for COCO polygon parsing + raw-dir path + optional network test.
library/tests/unit/scripts/test_prepare_food41.py	Adds unit tests for Food-101 root detection, split resolution, dataset build, and CLI failure mode.
library/tests/unit/scripts/test_prepare_roboflow_hf_datasets.py	Adds dataset registry entries for new Roboflow HF datasets.
library/tests/unit/scripts/test_roboflow_hf_helper.py	Extends tests for repo-relative parquet paths and optional segmentation polygons.
library/uv.lock	Updates lockfile for new extra (benchmark) and related dependency resolution changes.

[github-actions]

🐳 Docker image sizes

Device	Size
xpu	10791.5 MB (10.54 GB)
cuda	10198.1 MB (9.96 GB)
cpu	4002.9 MB (3.91 GB)

[Copilot]

Pull request overview

Copilot reviewed 35 out of 36 changed files in this pull request and generated 8 comments.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 95.57522% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
library/src/getitune/benchmark/catalog.py	96.66%	2 Missing ⚠️
library/src/getitune/benchmark/dataset_helpers.py	94.11%	2 Missing ⚠️
library/src/getitune/benchmark/manifest.py	88.88%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

📊 Test coverage report

Metric	Coverage
Lines	60.0%
Functions	56.7%
Branches	51.5%
Statements	59.6%

[Copilot]

Pull request overview

Copilot reviewed 39 out of 41 changed files in this pull request and generated 7 comments.

[AlbertvanHouten]

Why is this change needed? We do not have any versioning system for datasets so we would need to manually delete/update existing datasets on a benchmark runner.

[kprokofi]

it is the same. I changed this: _HF_RESOLVE = "https://huggingface.co/datasets/{repo}/resolve/{revision}/{path}"

So, at the end nothing should be changed for existing datasets.

To unify with other datasets that don't consist "data" in path, like pothole and few other