Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Docs

- **README rewritten as a concise landing page** (#161): Trimmed the root README from 707 to 258 lines by moving deep reference material (profiling tables, extended config/usage recipes, tips) into `docs/` and linking out. Replaced the stale ASCII architecture block and unreferenced `docs/img` PNGs with accurate inline Mermaid diagrams for the layered architecture, build→run→report pipeline, and deployment-target inference; added matching diagrams to `docs/deployment.md` and `docs/README.md`. Also corrects numerous stale references across docs: `--csv-file` → `--csv-file-path`/`--file`, missing `database` command flags (`--unique-key`/`-k`, `--batch-size`, `--no-upsert`, `--no-index`, `--dry-run`, `MONGO_AUTH_SOURCE`/`MONGO_TIMEOUT_MS`), wrong `run --output`/`--tools-config` defaults, `megatron` → `megatron-lm` launcher name, fabricated `timeout_multiplier`/`service_account` config keys, missing Kubernetes/SLURM `additional_context` keys, `DOCKER_CONFIG`/`MAD_SKIP_DOCKER_LOGIN` documentation, and corrected SGLang Disaggregated minimum node counts/split formula for SLURM vs. Kubernetes.

## [2.2.0] - 2026-08-12

### Added

- **`slurm.skip_gpus_directive` opts out of `#SBATCH --gpus-per-node`** (#163): Clusters that don't advertise GPU GRES reject any job script carrying `--gpus-per-node`, failing submission before launch. Setting `slurm.skip_gpus_directive: true` omits the directive from the generated `job.sh.j2`, relying on `exclusive`/`nproc_per_node` instead. Defaults to `false` (directive still emitted).

- **Reuse existing `docker login` (OAT) instead of requiring `credential.json`** (#168): madengine previously required credentials to be duplicated into `credential.json`/env vars even when the machine already had a working `docker login` (e.g. an organization access token). `has_ambient_docker_auth()` (`core/auth.py`) now reads `${DOCKER_CONFIG:-~/.docker}/config.json` the same way the Docker CLI does — covering `auths`, `credHelpers`, and `credsStore` — so an existing login is reused and blank placeholder credentials (`{"username": "", "password": ""}`) never override or break it. `docker build --pull` now only logs in to the base image's registry when there's no existing login to reuse. Base-image pull failures also distinguish `insufficient_scope` (authorization — token needs wider repo scope) from `unauthorized`/`authentication required` (login problem), and `explain_registry_denial()` now names the actual failing registry (e.g. `ghcr.io`) in its `docker login`/`credential.json` suggestions instead of always assuming Docker Hub. Documented in `docs/configuration.md` under a new "Registry Authentication" section.

### Fixed

- **Generated SLURM job script portable across clusters** (#163): The rendered `job.sh.j2` made several assumptions that don't hold on every cluster. It now inherits the submitter's PATH explicitly — a batch job isn't guaranteed to keep it (a site can default `sbatch` to `--export=NONE`, and `module load` can rewrite it) — by re-exporting `$HOME/.local/bin` and the directory `madengine` was resolved from at submission time (`SlurmDeployment._submission_bin_dir()`), so a job that passed the pre-submission availability check no longer fails on the compute node with "madengine not found in PATH". The pre-submission `madengine --version` probe's timeout was also raised from 5s to 600s, since a cold interpreter start off shared/NFS storage routinely exceeded 5s on a healthy environment. The single-node shared-workspace probe now reads the filesystem type alone via `df --output=fstype` (with an `awk`-based fallback for pre-8.21 coreutils) instead of grepping the whole `df -T` line, which previously classified a local disk mounted at a path like `/mnt/nfs-scratch` as shared storage; the type pattern also now matches `nfs4` (not just `nfs`) and adds `beegfs`/`panfs`. Separately, `RunOrchestrator`'s informational `rocm-libs` package-manager query (`apt`/`yum`/`zypper`/`tdnf`) is now wrapped in `timeout 10` so a node prompting interactively (e.g. `yum` asking to import a repo GPG key) can no longer hang an entire multi-node run. Covered by a new `tests/unit/test_slurm_job_template.py`.

- **`multiple_results` preserved in local-image manifest** (#166): `MAD_CONTAINER_IMAGE` (local-image) mode built a synthetic manifest that omitted the model's `multiple_results` field from `models.json`. Without it, `ContainerRunner` never set `MAD_OUTPUT_CSV` or copied the perf CSV out of the container, instead falling back to scraping the run log for a `"performance: NUMBER METRIC"` line — reporting `FAILURE` even when the model produced valid perf-CSV results. `RunOrchestrator` now carries `multiple_results` through to the manifest it synthesizes for local-image runs.

- **GPU tool detection honors non-default ROCm install paths and PATH** (#168): `amd_smi_utils.py`, `rocm_smi_utils.py`, and `gpu_info_profiler.py` hardcoded `/opt/rocm`; they now honor `$ROCM_PATH` (falling back to `/opt/rocm`). `gpu_info_pre.sh` detects `nvidia-smi`/`rocm-smi`/`amd-smi` via `command -v` instead of a fixed binary path, and calls `rocminfo` only when it's available instead of failing the pre-script when it's absent. The rpd tracer's `LD_LIBRARY_PATH` is now ROCm-path-aware, and `rpd2tracing.py` failures fall back to saving the raw `trace.rpd` instead of losing the trace.

## [2.1.3] - 2026-07-15

### Added
Expand Down