Adam/h3 rvm posttraining

.agents/lessons/2026-08-14_h3-dmd2-training-preflight-rules.md

@@ -0,0 +1,120 @@
+---
+date: 2026-08-14
+experiment: feat/h3-dmd2-vsa — data-free DMD2 distillation of MiniMax-H3 (33B, video+audio) on NVL72 GB200
+category: training
+severity: critical
+---
+
+# Preflight Rules for Large-Model Distillation Runs (distilled from the H3 DMD2 v1–v5 lineage)
+
+Every rule below is a real failure we shipped, diagnosed, and fixed across
+five production launches. Rules are grouped; each states the check, the
+failure it prevents, and where it is now enforced in code (if it is).
+
+## Numerics
+
+1. **Master weights must be fp32.** In-place optimizer steps on bf16 params
+   round away updates below ~half an ulp of each weight's magnitude:
+   O(1)-magnitude norm gains froze **bit-identically for 1000 steps**
+   (205/211 tensors) at lr 2e-6, silently capping convergence. Enforced:
+   `fastvideo/train/trainer.py::_verify_master_weight_precision` and
+   `fastvideo/training/training_utils.py::verify_master_weight_precision`
+   hard-fail at train start (opt-outs:
+   `training.model.allow_low_precision_master_weights`,
+   `FASTVIDEO_ALLOW_LOW_PRECISION_MASTER_WEIGHTS=1`).
+2. **Optimizer second moments must be fp32 regardless of param dtype.**
+   `torch.zeros_like(p)` inherits the starved dtype; bf16 `exp_avg_sq` loses
+   g² increments below ~0.4% of its running magnitude. Enforced in
+   `AdamWBeta1Zero` (fp32 state + fp32 update math for non-fp32 params).
+3. **Do not copy inference precision defaults into training configs.**
+   `PipelineConfig.dit_precision` defaults to bf16 (inference); training
+   YAMLs that copy it inherit rule-1's failure. `dit_precision: fp32` costs
+   only sharded-master memory — FSDP `param_dtype=bf16` keeps compute speed
+   (measured: no step-time change at 33B).
+4. **Post-hoc freeze check (cheap, mapping-free):** base checkpoints are
+   stored bf16, so any fp32-updated tensor is almost surely no longer
+   exactly bf16-representable. Census a checkpoint's norm params for
+   `x != bf16(x)` fractions — near-zero fractions mean weights are not
+   moving. Verified on the v5 lineage: at checkpoint-500 with fp32 masters,
+   210/210 student norm tensors moved (median 99.9% of elements), vs
+   205/211 bit-frozen after 1000 bf16 steps.
+
+## Schedules and shifts
+
+5. **Recompute every timestep-space constant through the model's actual
+   shift map** (`sigma' = s*t / (1 + (s-1)t)`). The Wan-copied ladder
+   `[1000,757,522]` under H3's video shift 12 produced sigma
+   {1.0, 0.974, 0.929} — a near-no-op middle step and an 86% final jump.
+   Re-spaced to `[1000,667,333]` (uniform base-t, the same design rule the
+   Wan recipe uses pre-warp).
+6. **Check supervision *density* in sigma space, not t space.** Uniform-t
+   sampling under shift 12 put 57% of score-model draws at sigma_v > 0.9 and
+   ~0% below 0.2. Fix: `method.score_timestep_shift` (sample uniform in
+   shifted-sigma, invert to base-t). Verify any new sampler with a quantile
+   census before launching.
+7. **Shared timestep-ratio bounds hit modalities asymmetrically.** With
+   min_ratio 0.02, video's supervision floor was sigma 0.197 while audio's
+   was 0.057 (shift 12 vs 3). Derive per-modality sigma floors from the
+   ratios; H3 uses 0.005.
+8. **Know whether a step list is consumed in base-t or shifted space.**
+   `warp_denoising_step` maps the list through the (video) scheduler's
+   shifted grid; H3's adapter *also* shifts per modality internally —
+   enabling warp double-shifts. Wan needs warp on; H3 needs it off.
+
+## Packed multi-modality
+
+9. **Never take a single mean loss over a packed multi-modality sequence.**
+   H3 packs video:audio at ~270:1 elements; a packed mean silently trains
+   only video. Compute per-modality losses via `modality_slices()` and log
+   each stream (`*_video`, `*_audio`) separately.
+
+## Distillation specifics
+
+10. **Check whether the teacher is guidance-distilled before configuring
+    CFG.** H3's released checkpoint rejects CFG at inference; correct
+    setting is `real_score_guidance_scale: 1.0`, and DMD2 then skips the
+    unconditional teacher forward entirely (free speedup).
+11. **Read DMD scalars as game state, not quality.** Generator loss =
+    normalized critic–teacher disagreement at student samples (healthy band
+    ~0.2–1.0; it *rises* from ~0 as the critic specializes away from its
+    teacher init). Critic flow-matching loss = tracking meter (falling =
+    keeping up). The 10x-LR blowup signature: critic loss exploding (5.2)
+    plus student grad-norms pinned above clip (1.8–2.5 vs 1.0) — a run that
+    reports 10x LR but delivers ~4x, into a broken critic.
+12. **Grad-norm vs clip tells you the LR you actually delivered.** Isolated
+    spikes above clip that mean-revert are normal equilibration; sustained
+    saturation means the configured LR is a lie.
+13. **Sampler A/Bs need a teacher control and matched seeds.** The
+    stochastic-vs-deterministic verdict on a student is meaningless without
+    knowing the teacher's own behavior under both hop rules (teacher x0
+    overshoot at sigma≈1, std 2.33, dominated ours). File-size heuristics
+    on encoded video are not a quality signal — high bitrate can be noise.
+
+## Run ops
+
+14. **Requeue-safe by default:** `resume_from_checkpoint: latest` +
+    `sbatch --requeue`. DCP restores optimizer LRs over the YAML — use
+    `reset_lr_on_resume: true` for LR-change experiments.
+15. **Budget checkpoint disk before launch:** save size × retention vs
+    filesystem free. fp32 masters tripled saves to 741 GB (measured; student+critic params + fp32 second moments); lustre at 98%
+    would have killed the run at its 4th save. `checkpoints_total_limit`
+    is part of the launch math, not a detail.
+16. **Verify W&B auth (`wandb.Api().viewer` with the job's $HOME) before
+    every launch**; the tracker now degrades online→offline→none instead of
+    crashing 32 GPUs on a rotated key. Confirm step-0 media actually
+    uploaded — the uploader has died on the first video burst before.
+17. **Unique `run_name` per launch attempt.** Three runs sharing one name
+    made W&B show another run's config (the "DP=12" confusion).
+18. **Cluster quirks (Slinky/GB200):** compute pods have no /home (export
+    HOME=lustre in every sbatch; submit with a lustre --chdir); PENDing
+    jobs are what make the autoscaler create nodes (warmup-ladder + race
+    for multi-node); released nodes deregister in seconds; boot watchdog +
+    NCCL env pins for the silent IB comm-init wedge; `scontrol release`
+    after "launch failed requeued held"; Slurm requeues silently truncate
+    stdout (check elapsed vs log).
+19. **Exports and deletions:** cross-user hardlinks are blocked
+    (fs.protected_hardlinks) — use `--link-base` symlink exports (~66 GB vs
+    465 GB); always clear stale `*.safetensors.index.json` when rewriting a
+    module dir (a leftover base index shadows the fresh weights); before any
+    rm of run artifacts, verify no symlinks/hardlinks point into originals
+    and that the target is regenerable.

HANDOFF-h3-dmd2-vsa.md

@@ -0,0 +1,88 @@
+# Handoff: VSA-H3 DMD2 overfit — run directly on tray 1795370
+
+State as of 2026-08-09. Everything is staged on the tray; only the launch is
+left (my login-node srun path was blocked by per-user thread exhaustion —
+running inside the tray's container sidesteps it entirely).
+
+## The experiment
+
+Single-sample DMD2 overfit on `c17_v2` (your prompt-rewrite v2 generation,
+1344×768 / 5.2 s / with audio), parity-first: **student on the VSA-H3 backend
+at sparsity 0.0** (parity-proven ≤2e-4 vs dense in unit tests) vs **teacher +
+critic on FLASH_ATTN with FA4**. 3-step schedule `[1000, 757, 522]`,
+`generator_update_interval: 5` (TTUR: critic every step, student every 5th),
+4 GPUs, sp=4, pure-FSDP sharding (hsdp 1×4), all three models
+gradient-checkpointed, batch 1, 2000 steps, ckpt+validation every 250.
+
+## What is already staged
+
+- **Code**: `/mnt/fv-h3ssim` on branch `h3-dmd2-vsa` (= `feat/h3-dmd2-vsa`
+  @ d81f1f166 = DMD2-for-H3 + PR #1695 VSA backend + per-role backend wiring
+  + wandb credential guard). Also pushed to origin as `feat/h3-dmd2-vsa`.
+- **Data**: `/mnt/h3-dmd2-overfit/data/data_00000.parquet` — one preprocessed
+  t2va record (video+audio VAE latents + text embedding 225×5120).
+- **Validation prompts**: `/mnt/h3-dmd2-overfit/c17_validation.json` (the c17
+  v2 prompt, 3-step sampling) — the YAML may still point at the Wan
+  placeholder; step 2 below fixes it idempotently.
+- **Config**: `examples/train/configs/distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml`
+- Raw inputs (for reference): `/mnt/h3-dmd2-overfit/c17_v2.mp4`, `c17_v2_prompt.txt`.
+
+## Run it (on the tray)
+
+```bash
+# 0. enter the persistent container (created by my earlier runs)
+docker exec -it fv-dev-1795370-0 bash
+# (if it's gone: ~/docker.sh style run with /home/scratch.willlin_ent -> /mnt)
+
+# 1. clean any leftover trainer from my attempts + confirm GPUs are free
+ps -eo pid,args | grep entrypoint.train | grep -v grep   # kill any PIDs listed
+nvidia-smi --query-gpu=memory.used --format=csv,noheader  # expect ~0 MiB x4
+
+# 2. env + config fix (idempotent)
+cd /mnt/fv-h3ssim
+export PYTHONPATH=/mnt/studio8/pyoverride   # routes 'import fastvideo' to THIS checkout
+export FASTVIDEO_FA4=1                      # teacher/critic FLASH_ATTN -> FA4 path
+export WANDB_API_KEY=8d9f4b39abd68eb4e29f6fc010b7ee71a2207cde
+sed -i "s|dataset_file: examples/training/finetune/Wan2.1-Fun-1.3B-InP/crush_smol/validation.json|dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json|" \
+  examples/train/configs/distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml
+
+# 3. launch (interactive; or wrap in setsid nohup ... & for detach)
+torchrun --nproc_per_node 4 -m fastvideo.train.entrypoint.train \
+  --config examples/train/configs/distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml \
+  2>&1 | tee /mnt/h3-dmd2-overfit/logs/train.log
+```
+
+wandb: project `h3-dmd2-vsa`, run `dmd2_vsa0_overfit` — the URL prints within
+the first seconds of trainer init (`https://wandb.ai/...`). Logged per step:
+total/generator/fake-score loss, update_student flag, step_time_sec,
+vsa_sparsity.
+
+## What to expect / what parity means
+
+- Trio load: ~3×(33B DiT) FSDP-sharded + text encoder; first load on this
+  tray is warm for the text encoder (preprocess touched it) but cold for the
+  3 transformer copies — expect ~10-20 min before step 1.
+- Healthy signals: both losses finite from step 1; fake-score loss drops
+  fast (critic fitting one sample); generator loss moves every 5th step.
+- Parity claim to check: with sparsity 0.0 the student's VSA-H3 path should
+  train indistinguishably from a dense student (the sparsity-0 mask selects
+  every tile; kernel parity ≤2e-4). If it diverges/NaNs where a dense run
+  wouldn't, that's a VSA-H3-under-autograd bug — grab
+  `/mnt/h3-dmd2-overfit/logs/train.log` and the wandb run.
+- Next knob after parity holds: `training.vsa.sparsity` (e.g. 0.5 → 0.875
+  → 0.9) in the YAML — student-only; teacher/critic stay dense by
+  construction (`models.*.attention_backend`).
+
+## Known traps (all pre-paid)
+
+- `PYTHONPATH=/mnt/studio8/pyoverride` is mandatory — without it,
+  `import fastvideo` resolves to `/mnt/FastVideo` (the maintainer's checkout)
+  via the venv's editable-install finders, and cwd `/mnt/FastVideo` shadows too.
+- Don't set `FASTVIDEO_ATTENTION_BACKEND` globally — per-role backends come
+  from the YAML (`models.student/teacher/critic.attention_backend`).
+- The tile-buffer autograd fix is in this branch (grad paths get fresh
+  buffers); if you see "modified by an inplace operation" at backward, you're
+  on a stale checkout — `git -C /mnt/fv-h3ssim log --oneline -1` should show
+  d81f1f16.
+- Checkpoints land in `outputs/minimax_h3_dmd2_vsa0_overfit/` under the
+  checkout (NFS); validation videos under the same tree per callback config.

HANDOFF-slurm-32xgb200.md

@@ -0,0 +1,120 @@
+# Handoff: MiniMax-H3 data-free DMD2 on 8x 4xGB200 (Slurm)
+
+State as of 2026-08-11 on branch `feat/h3-dmd2-vsa`. Everything needed for
+the 32-GPU run is committed; what remains is cluster-side: fill in the
+`CLUSTER:` knobs in the sbatch, swap in the real dataset, and submit.
+
+## What this run is
+
+Data-free DMD2 distillation (`rollout_mode: simulate` — the student builds
+its own trajectory from pure noise through `[1000, 757, 522]`; no video/audio
+latents consumed, text conditioning only) of the 33B dual-modality H3 DiT.
+All three roles (student/teacher/critic) on dense FLASH_ATTN with FA4.
+**Global batch 64** = 8 data-parallel groups (one per tray) x
+`gradient_accumulation_steps: 8`.
+
+- Config: `examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_32gpu.yaml`
+- Launcher: `examples/train/slurm/dmd2_32xgb200.sbatch`
+- Docs: `examples/train/configs/distribution_matching/minimax_h3/README.md`
+
+## Everything below is already validated on a single 4xGB200 node
+
+- The exact simulate recipe trained stably (wandb `h3-dmd2-vsa/z2z91ruf`):
+  TTUR cadence correct, generator loss settles into the expected adversarial
+  0.2–1.0 band, no NaN/OOM. ~10 s/it critic steps, ~20 s/it generator steps
+  (every 5th) at 1344x768x124f, sp=4.
+- Effective-batch-2 via gradient accumulation over two distinct videos
+  (wandb `58dr571y`, 1800+ steps): both modalities' losses decline
+  monotonically (video 0.128->0.092, audio 0.179->0.053 by step 1000).
+  Accumulation is the batching mechanism — H3's packed pipeline is batch-1
+  per forward by design (variable caption lengths cannot stack).
+- Memory: three 33B roles fit at sp=4 / HSDP shard=4 with full gradient
+  checkpointing and `betas: [0.0, 0.999]` (buffer-free AdamW). Do NOT try
+  sp=2 or fewer than 4 GPUs per model replica: the backward working set
+  alone OOMs 184 GiB cards (measured 178 GiB before optimizer states exist).
+- Triton VSA kernel backward fix, bf16 autocast boundaries, validation
+  offload fix, sigma-exact validation — all on the branch (see git log).
+
+## Cluster pre-flight
+
+1. **Code**: checkout `feat/h3-dmd2-vsa` on the shared FS; venv with the
+   repo's CUDA install. `run.sh` sets PYTHONPATH to the repo root and cd's
+   into it — on trays that carry `/mnt/studio8/pyoverride`, make sure it is
+   NOT in PYTHONPATH or `import fastvideo` resolves to the wrong checkout.
+2. **Weights**: `/mnt/models/MiniMax-H3` must be visible from compute nodes
+   (all three roles `init_from` it).
+3. **Data**: the config ships pointing at the 2-row smoke-test set
+   (`/mnt/h3-dmd2-overfit/data_bs2`) — fine for the first submission, swap
+   `training.data.data_path` (and the validation `dataset_file`) for the
+   real preprocessed t2va parquet dir before a production run. Simulate mode
+   only reads text conditioning from the rows.
+4. **sbatch knobs**: partition/account (marked `CLUSTER:`), and if trays
+   require containers, wrap the srun payload (pyxis
+   `--container-image/--container-mounts=/mnt:/mnt,/dev/shm`) — hooks are
+   marked in the script.
+5. **W&B**: `~/.netrc` on a home visible from compute nodes, or export
+   `WANDB_MODE=offline` at submit time.
+6. **Login-node limits**: a previous srun attempt died to per-user thread
+   exhaustion on the login node — submit with `sbatch` (batch daemon runs
+   the script on the first compute node), don't launch interactive `srun`
+   pipelines from the login shell.
+
+## Launch
+
+```bash
+sbatch examples/train/slurm/dmd2_32xgb200.sbatch                 # 8 trays / 32 GPUs
+sbatch --nodes=2 examples/train/slurm/dmd2_32xgb200.sbatch       # scaled smoke test
+CONFIG=<other.yaml> sbatch examples/train/slurm/dmd2_32xgb200.sbatch
+```
+
+The script derives world size / rendezvous / `hsdp_replicate_dim` from the
+actual allocation, so a smaller `--nodes` shrinks the DP width (and global
+batch) consistently. Recommended sequence: `--nodes=1` (reproduces the
+validated local run), then `--nodes=2`, then the full 8.
+
+## What to expect
+
+- **First boot per node**: ~10-13 min per 33B role from cold NFS (~35 min
+  worst case for the trio, plus text encoder/VAEs) — then each node writes
+  its weight cache to `/dev/shm/fastvideo-wcache`
+  (`FASTVIDEO_WEIGHT_SHARD_CACHE_PER_NODE=1` is set by the sbatch so every
+  tray caches, not just tray 0). **Relaunches: ~2-4 s per role.** Note
+  `/dev/shm` may be purged by node-local cleaners — a purged cache degrades
+  to a full load and rewrites itself, never fails the run.
+- **Step time**: per-tray compute is identical to the validated 4-GPU run;
+  with accum 8, expect roughly 8 x (10 s + 2 s amortized generator surcharge)
+  ≈ 90-100 s per optimizer step, ~4000 steps ≈ 4.5 days. Cut
+  `max_train_steps` or validation frequency to taste.
+- **Logs**: one dir per node under `examples/train/logs/<jobid>-node<k>/`;
+  wandb project `h3-dmd2-vsa`, run `dmd2_simulate_fa4_32gpu` (rank 0 only).
+- **Checkpointing** is ON every 500 steps (DCP; large — ~120 GiB+ per save
+  for two trainable roles). Point `output_dir` somewhere with headroom or
+  set `training_state_checkpointing_steps: 0` to disable.
+- **Validation** every 250 steps samples every prompt in the validation
+  json at the training sigmas; with the smoke-test json that is 2 videos,
+  ~3 min.
+
+## Known traps (all pre-paid on the branch)
+
+- Don't set `FASTVIDEO_ATTENTION_BACKEND` globally — per-role backends come
+  from the YAML.
+- `FASTVIDEO_FA4=1` is required (sbatch sets it); without it FLASH_ATTN
+  falls back to a slower path.
+- `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is required (sbatch
+  sets it); without it fragmentation OOMs appear after a few steps.
+- If a VSA student is used later (`attention_backend: VIDEO_SPARSE_ATTN_H3`
+  + `training.vsa.sparsity`): the Triton backward bugfix is on this branch —
+  do not run VSA training from an older checkout. Sparsity ~0.9+ matches
+  dense wall-clock in training and ~2x in validation sampling; the win is
+  inference-side.
+- NCCL fabric env (IB HCA / socket ifname) is cluster-specific — hook in the
+  sbatch.
+
+## If something breaks
+
+Grab the failing node's `examples/train/logs/<jobid>-node<k>/` log and the
+wandb run. The historically likely failure classes and their signatures:
+OOM ledger lines (memory — check betas are [0.0, 0.999] and grad ckpt full),
+`attempting to assign a gradient with device type` (a module was CPU-offloaded
+mid-training — should be fixed, see validation.py), rendezvous timeout at
+boot (MASTER_ADDR/port reachability between trays).