From 22c068e2b6fb81d4af29311a8864f992d338868e Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Wed, 12 Aug 2026 08:21:51 -0400 Subject: [PATCH 1/3] Updated CHANGELOG for v2.2.0 release --- CHANGELOG.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d35f2601..26f03b2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [2.2.0] - 2026-08-12 + +### Added + +- **Reuse existing `docker login` (OAT) instead of requiring `credential.json`** (#168): madengine previously required credentials to be duplicated into `credential.json`/env vars even when the machine already had a working `docker login` (e.g. an organization access token). `has_ambient_docker_auth()` (`core/auth.py`) now reads `${DOCKER_CONFIG:-~/.docker}/config.json` the same way the Docker CLI does — covering `auths`, `credHelpers`, and `credsStore` — so an existing login is reused and blank placeholder credentials (`{"username": "", "password": ""}`) never override or break it. `docker build --pull` now only logs in to the base image's registry when there's no existing login to reuse. Base-image pull failures also distinguish `insufficient_scope` (authorization — token needs wider repo scope) from `unauthorized`/`authentication required` (login problem), and `explain_registry_denial()` now names the actual failing registry (e.g. `ghcr.io`) in its `docker login`/`credential.json` suggestions instead of always assuming Docker Hub. Documented in `docs/configuration.md` under a new "Registry Authentication" section. + +### Fixed + +- **`multiple_results` preserved in local-image manifest** (#166): `MAD_CONTAINER_IMAGE` (local-image) mode built a synthetic manifest that omitted the model's `multiple_results` field from `models.json`. Without it, `ContainerRunner` never set `MAD_OUTPUT_CSV` or copied the perf CSV out of the container, instead falling back to scraping the run log for a `"performance: NUMBER METRIC"` line — reporting `FAILURE` even when the model produced valid perf-CSV results. `RunOrchestrator` now carries `multiple_results` through to the manifest it synthesizes for local-image runs. + +- **GPU tool detection honors non-default ROCm install paths and PATH** (#168): `amd_smi_utils.py`, `rocm_smi_utils.py`, and `gpu_info_profiler.py` hardcoded `/opt/rocm`; they now honor `$ROCM_PATH` (falling back to `/opt/rocm`). `gpu_info_pre.sh` detects `nvidia-smi`/`rocm-smi`/`amd-smi` via `command -v` instead of a fixed binary path, and calls `rocminfo` only when it's available instead of failing the pre-script when it's absent. The rpd tracer's `LD_LIBRARY_PATH` is now ROCm-path-aware, and `rpd2tracing.py` failures fall back to saving the raw `trace.rpd` instead of losing the trace. + ## [2.1.3] - 2026-07-15 ### Added From 7e5f3c21bc1772d29a2ae4e6c9eed180e8388f57 Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Wed, 19 Aug 2026 10:27:27 -0500 Subject: [PATCH 2/3] docs(changelog): add v2.2.0 entries for #163 Reflects fix(slurm): make the generated job script portable across clusters (c5b016f), which landed after the CHANGELOG's v2.2.0 section was first written. --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 26f03b2d..9b549773 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,10 +11,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **`slurm.skip_gpus_directive` opts out of `#SBATCH --gpus-per-node`** (#163): Clusters that don't advertise GPU GRES reject any job script carrying `--gpus-per-node`, failing submission before launch. Setting `slurm.skip_gpus_directive: true` omits the directive from the generated `job.sh.j2`, relying on `exclusive`/`nproc_per_node` instead. Defaults to `false` (directive still emitted). + - **Reuse existing `docker login` (OAT) instead of requiring `credential.json`** (#168): madengine previously required credentials to be duplicated into `credential.json`/env vars even when the machine already had a working `docker login` (e.g. an organization access token). `has_ambient_docker_auth()` (`core/auth.py`) now reads `${DOCKER_CONFIG:-~/.docker}/config.json` the same way the Docker CLI does — covering `auths`, `credHelpers`, and `credsStore` — so an existing login is reused and blank placeholder credentials (`{"username": "", "password": ""}`) never override or break it. `docker build --pull` now only logs in to the base image's registry when there's no existing login to reuse. Base-image pull failures also distinguish `insufficient_scope` (authorization — token needs wider repo scope) from `unauthorized`/`authentication required` (login problem), and `explain_registry_denial()` now names the actual failing registry (e.g. `ghcr.io`) in its `docker login`/`credential.json` suggestions instead of always assuming Docker Hub. Documented in `docs/configuration.md` under a new "Registry Authentication" section. ### Fixed +- **Generated SLURM job script portable across clusters** (#163): The rendered `job.sh.j2` made several assumptions that don't hold on every cluster. It now inherits the submitter's PATH explicitly — a batch job isn't guaranteed to keep it (a site can default `sbatch` to `--export=NONE`, and `module load` can rewrite it) — by re-exporting `$HOME/.local/bin` and the directory `madengine` was resolved from at submission time (`SlurmDeployment._submission_bin_dir()`), so a job that passed the pre-submission availability check no longer fails on the compute node with "madengine not found in PATH". The pre-submission `madengine --version` probe's timeout was also raised from 5s to 600s, since a cold interpreter start off shared/NFS storage routinely exceeded 5s on a healthy environment. The single-node shared-workspace probe now reads the filesystem type alone via `df --output=fstype` (with an `awk`-based fallback for pre-8.21 coreutils) instead of grepping the whole `df -T` line, which previously classified a local disk mounted at a path like `/mnt/nfs-scratch` as shared storage; the type pattern also now matches `nfs4` (not just `nfs`) and adds `beegfs`/`panfs`. Separately, `RunOrchestrator`'s informational `rocm-libs` package-manager query (`apt`/`yum`/`zypper`/`tdnf`) is now wrapped in `timeout 10` so a node prompting interactively (e.g. `yum` asking to import a repo GPG key) can no longer hang an entire multi-node run. Covered by a new `tests/unit/test_slurm_job_template.py`. + - **`multiple_results` preserved in local-image manifest** (#166): `MAD_CONTAINER_IMAGE` (local-image) mode built a synthetic manifest that omitted the model's `multiple_results` field from `models.json`. Without it, `ContainerRunner` never set `MAD_OUTPUT_CSV` or copied the perf CSV out of the container, instead falling back to scraping the run log for a `"performance: NUMBER METRIC"` line — reporting `FAILURE` even when the model produced valid perf-CSV results. `RunOrchestrator` now carries `multiple_results` through to the manifest it synthesizes for local-image runs. - **GPU tool detection honors non-default ROCm install paths and PATH** (#168): `amd_smi_utils.py`, `rocm_smi_utils.py`, and `gpu_info_profiler.py` hardcoded `/opt/rocm`; they now honor `$ROCM_PATH` (falling back to `/opt/rocm`). `gpu_info_pre.sh` detects `nvidia-smi`/`rocm-smi`/`amd-smi` via `command -v` instead of a fixed binary path, and calls `rocminfo` only when it's available instead of failing the pre-script when it's absent. The rpd tracer's `LD_LIBRARY_PATH` is now ROCm-path-aware, and `rpd2tracing.py` failures fall back to saving the raw `trace.rpd` instead of losing the trace. From 6bd93878dfd846d02a552cabb419718ac024515c Mon Sep 17 00:00:00 2001 From: Stephen Shao Date: Wed, 19 Aug 2026 20:41:50 -0500 Subject: [PATCH 3/3] docs(changelog): add entry for README rewrite (#161) --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9b549773..4e7e4428 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Docs + +- **README rewritten as a concise landing page** (#161): Trimmed the root README from 707 to 258 lines by moving deep reference material (profiling tables, extended config/usage recipes, tips) into `docs/` and linking out. Replaced the stale ASCII architecture block and unreferenced `docs/img` PNGs with accurate inline Mermaid diagrams for the layered architecture, build→run→report pipeline, and deployment-target inference; added matching diagrams to `docs/deployment.md` and `docs/README.md`. Also corrects numerous stale references across docs: `--csv-file` → `--csv-file-path`/`--file`, missing `database` command flags (`--unique-key`/`-k`, `--batch-size`, `--no-upsert`, `--no-index`, `--dry-run`, `MONGO_AUTH_SOURCE`/`MONGO_TIMEOUT_MS`), wrong `run --output`/`--tools-config` defaults, `megatron` → `megatron-lm` launcher name, fabricated `timeout_multiplier`/`service_account` config keys, missing Kubernetes/SLURM `additional_context` keys, `DOCKER_CONFIG`/`MAD_SKIP_DOCKER_LOGIN` documentation, and corrected SGLang Disaggregated minimum node counts/split formula for SLURM vs. Kubernetes. + ## [2.2.0] - 2026-08-12 ### Added