Adam/h3 rvm posttraining

[Abecid]

Purpose

Add a production-ready Reward-based Velocity Matching (RVM) post-training
path for the released four-forward FastH3 VSA checkpoint.

The goal is to improve prompt adherence, motion quality, and visual preference
while preserving FastH3's deployed four-step sampler, sparse-attention policy,
and joint video/audio behavior. This PR follows the published RVM objective and
reward recipe rather than introducing a new diffusion-RL algorithm.

There is no linked issue.

Changes

Paper-faithful FastH3 RVM

• Added on-policy endpoint RVM for FastH3:
  • behavior rollouts use the released four-step schedule
    1000, 750, 500, 250 -> 0;
  • CFG remains disabled (guidance_scale=1, conditioning dropout 0);
  • rollouts and training use VIDEO_SPARSE_ATTN_H3 at 90% sparsity;
  • generated endpoints are analytically noised and trained with the native
    flow target epsilon - x0.
• Matched the published reward normalization:
  • subtract the mean within each prompt's K candidates;
  • divide by one standard deviation computed over the complete rollout
    collection, including all data-parallel replicas;
  • apply the published 0.1 scale and signed clipping.
• Sample the RVM regression time continuously with
  t ~ Uniform(0, 1). Deployment-grid training remains an explicit ablation;
  behavior generation always stays four-step.
• Implemented the signed update through a detached nonnegative MSE target, which
  yields the intended signed RVM gradient without exposing the optimizer to an
  unbounded negative scalar loss.

H3 model and audio handling

• Added a packed video/audio H3 adapter with independent modality reductions and
  native video/audio scheduler shifts (12 and 3).
• Train only a new LoRA on to_q, to_k, to_v, and to_out; the released
  35B backbone and VSA compression gate remain frozen.
• Apply reward only to the video slice.
• Keep exact unanchored RVM as the reference configuration and provide isolated
  audio-only and full function-space anchor ablations.
• Obtain anchor predictions by disabling the quality LoRA in place, avoiding a
  second 35B reference model.

Reward stack and diagnostics

Implemented the public RVM video reward mixture:

1.5 * VideoAlign text alignment
1.0 * VideoAlign motion quality
0.1 * HPSv3 general preference
0.1 * HPSv3 prompt-conditioned top-30%-frame preference
0.7 * pretrained RAFT dynamic tracking

Also added:

• Transformers 5 compatibility for the pinned VideoAlign/HPSv3 reward models;
• bounded VAE and HPSv3 scoring batches for H100 memory;
• per-component reward logging;
• batch-global and prompt-group reward statistics;
• advantage magnitude/clipping diagnostics;
• gradient norm/clipping diagnostics;
• continuous training-time statistics;
• unclipped RAFT motion and saturation diagnostics.

Data, evaluation, and experiment scripts

• Added deterministic download, deduplication, split, H3 formatting, and
  Qwen3-VL embedding preprocessing for the DanceGRPO/VidProM prompt bank.
• Added one-GPU correctness, 8-GPU topology, LR, anchor, medium-scale, resume,
  checkpoint, LoRA export, and inference scripts.
• Evaluate every ceil(0.05 * max_train_steps) optimizer updates on at most 100
  fixed held-out prompts and seeds.
• Save per-checkpoint videos and reward metrics; training rollout reward and
  held-out reward are reported separately.
• Enforce a clean Git source tree and record HEAD and HEAD^{tree} before
  reported runs.
• Gate the 180-step / 23,040-endpoint campaign behind completed topology,
  reward, audio, resume, and export checks.

Provider-independent execution

• Moved dependency installation, model/reward download, dataset preparation,
  configs, hyperparameters, training orchestration, and result collection into
  ordinary repository scripts.
• Added 12_run_portable_smoke.sh, which runs the same one-/four-GPU smoke on
  any Docker host or cloud VM.
• Reduced modal_h3_rvm.py to a test-only transport wrapper: it allocates one or
  four GPUs, mounts volumes, checks out an exact Git ref, and invokes the
  portable runner.
• Production 8/16-H100 runs are launched from the normal custom-node scripts;
  they do not depend on Modal. The Modal file may be removed before merge
  without affecting the training path.

Test Plan

Static and unit checks

pre-commit run --all-files

python -m py_compile \
  fastvideo/train/methods/rl/rvm.py \
  fastvideo/train/methods/rl/rvm_faithful.py \
  fastvideo/train/methods/rl/rvm_local_metrics.py \
  fastvideo/train/methods/rl/common/rvm_utils.py \
  fastvideo/train/methods/rl/common/minimax_h3_rvm.py \
  fastvideo/train/methods/rl/rewards/media.py \
  fastvideo/train/methods/rl/rewards/dynamic_tracking.py \
  fastvideo/train/models/minimax_h3/minimax_h3_rvm.py \
  examples/train/rvm_h3/modal_h3_rvm.py

bash -n \
  examples/train/rvm_h3/00_install_current_env.sh \
  examples/train/rvm_h3/12_run_portable_smoke.sh \
  examples/train/rvm_h3/05_run_8gpu_topology_smoke.sh \
  examples/train/rvm_h3/05_run_8gpu_lr_sweep.sh \
  examples/train/rvm_h3/06_run_8gpu_anchor_sweep.sh \
  examples/train/rvm_h3/07_run_8gpu_scaleup_pilot.sh \
  examples/train/rvm_h3/07_run_8gpu_full.sh

pytest -q \
  fastvideo/tests/train/methods/test_rvm_utils.py \
  fastvideo/tests/train/methods/test_rvm_reward_diagnostics.py \
  fastvideo/tests/train/methods/test_modal_h3_rvm.py \
  fastvideo/tests/train/methods/test_minimax_h3_dmd2.py \
  fastvideo/tests/train/methods/test_rvm_configs.py \
  fastvideo/tests/inference/lora/test_merge_lora_math.py

git diff --check

One-/four-GPU integration gates

# One strict H100 compact topology smoke.
H3_RVM_MODAL_GPU_1='H100!' \
H3_RVM_MODAL_GPU_4='H100!:4' \
H3_RVM_MODAL_SECRETS='hf-adamlee00,wandb-adamlee00' \
  modal run examples/train/rvm_h3/modal_h3_rvm.py \
    --gpus 1 \
    --mode smoke \
    --max-steps 1 \
    --eval-prompts 1

# Four strict H100 production-geometry pilot.
H3_RVM_MODAL_GPU_1='H100!' \
H3_RVM_MODAL_GPU_4='H100!:4' \
H3_RVM_MODAL_SECRETS='hf-adamlee00,wandb-adamlee00' \
  modal run examples/train/rvm_h3/modal_h3_rvm.py \
    --gpus 4 \
    --mode pilot \
    --max-steps 10 \
    --eval-prompts 8

The same integration test can be run without Modal:

RVM_SKIP_CONDA=1 \
RVM_ARTIFACT_ROOT=/persistent/rvm_h3 \
RVM_SMOKE_RUN_ROOT=/persistent/runs \
RVM_SMOKE_GPUS=4 \
RVM_SMOKE_MODE=pilot \
RVM_SMOKE_MAX_STEPS=10 \
  bash examples/train/rvm_h3/12_run_portable_smoke.sh

Custom-node scale-up gates

export NUM_GPUS=8
export RVM_SP_SIZE=4

bash examples/train/rvm_h3/05_run_8gpu_topology_smoke.sh
bash examples/train/rvm_h3/05_run_8gpu_lr_sweep.sh

RVM_SELECTED_LR=<winner> \
  bash examples/train/rvm_h3/06_run_8gpu_anchor_sweep.sh

RVM_SELECTED_LR=<winner> \
RVM_SCALEUP_CONFIG=<winning-config> \
  bash examples/train/rvm_h3/07_run_8gpu_scaleup_pilot.sh

A 16-GPU run uses the same scripts with NUM_GPUS=16 and must first pass a
two-update topology smoke.

Test Results

Completed GPU validation before the latest RVM-fidelity refactor

Preflight:
- 29 focused tests passed
- all five configured reward models loaded and returned finite values

Public FastH3 inference:
- 480x832, 124 frames
- valid H.264 video and AAC stereo audio
- repeated fixed-seed runs were byte-identical

1x strict H100 80GB:
- compact 320x576x39, K=2
- rollout, reward, sparse-attention backward, Adam, checkpoint, and validation passed

4x strict H100 80GB:
- full 480x832x124 geometry
- SP4, K=4, rank-128 LoRA
- all five learned rewards
- two-step production pilot passed
- 34-step / 17-collection run completed in 3h18m20s
- checkpoints 17 and 34 saved
- no traceback, OOM, NCCL error, NaN, or Inf

The 34-step run was a runtime/non-collapse result, not a quality claim. On eight
fixed validation prompts, aggregate reward changed from 1.51039 to 1.46332
(-0.04707), with HPSv3 and dynamic tracking improving while VideoAlign TA/MQ
declined slightly.

The current head additionally changes reward normalization to batch-global
standard deviation, changes regression-time sampling to continuous uniform,
adds motion-saturation diagnostics, and refactors Modal into a thin wrapper.
Those latest changes have static/unit coverage but still require a fresh
one-H100 smoke and the 8-H100 SP4xDP2 topology gate before the long campaign.

Checklist

• I ran pre-commit run --all-files on the current PR head and fixed all issues
• I added or updated tests for my changes
• I updated documentation
• I considered GPU memory impact of my changes

For model/pipeline changes, also check:

• I verified SSIM regression tests pass on the current PR head
• I updated the support matrix if adding a new model — not applicable; this
  post-trains an existing FastH3 checkpoint

[mergify]

Pre-commit checks failed

Hi @Abecid, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

• yapf: yapf -i <file> (formatting)
• ruff: ruff check --fix <file> (linting)
• codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

[mergify]

Pre-commit checks failed

Hi @Abecid, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

• yapf: yapf -i <file> (formatting)
• ruff: ruff check --fix <file> (linting)
• codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

[mergify]

Pre-commit checks failed

Hi @Abecid, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

• yapf: yapf -i <file> (formatting)
• ruff: ruff check --fix <file> (linting)
• codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

[mergify]

Pre-commit checks failed

Hi @Abecid, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

• yapf: yapf -i <file> (formatting)
• ruff: ruff check --fix <file> (linting)
• codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

[mergify]

Pre-commit checks failed

Hi @Abecid, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

• yapf: yapf -i <file> (formatting)
• ruff: ruff check --fix <file> (linting)
• codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.