[perf]: MiniMax H3 - build the Qwen3-VL encoder only as far as it is read (-13.7 GB)

[KyleNeverGivesUp]

Purpose

MiniMax H3 conditions on one intermediate hidden state of its Qwen3-VL encoder
and reads nothing above it. The remaining 14 decoder layers and the final norm
are built, weight-loaded and then discarded: 13.7 GB in bf16, 21.7 percent of
the encoder.

Part of #1709.

The finding

MINIMAX_H3_TEXT_ENCODER_LAYER = 50 (pipelines/basic/minimax_h3/packing.py:31)
is consumed once:

# pipelines/basic/minimax_h3/stages/minimax_h3_conditioning.py:158,184
hidden_state_index = MINIMAX_H3_TEXT_ENCODER_LAYER
...
outputs.hidden_states[hidden_state_index].to(device=device, dtype=dtype),

Nothing else reads the encoder. last_hidden_state is produced at
models/encoders/minimax_h3_qwen3_vl.py:264 and has no consumer anywhere in the
tree. Meanwhile the stack is built in full:

# models/encoders/minimax_h3_qwen3_vl.py:230-232
self.layers = nn.ModuleList(
    MiniMaxH3Qwen3VLTextDecoderLayer(config, prefix=...)
    for index in range(config.num_hidden_layers))

Per layer, from the shipped config (hidden 5120, intermediate 25600, 64 q heads
and 8 kv heads at head_dim 128, no attention bias):

q 5120x8192                  41,943,040
k, v  2 x 5120x1024          10,485,760
o 8192x5120                  41,943,040
q_norm + k_norm                     256
gate/up/down 3 x 5120x25600 393,216,000
2 x RMSNorm                      10,240
                            -----------
                            487,598,336

14 layers = 6.83 B parameters = 13.7 GB in bf16.

Changes

num_hidden_layers_override on the arch config, defaulted to the tapped index.
This is the same field clip.py and siglip.py already carry, and CLIP ships it
set to 31, so the mechanism and the name are established rather than new.

min(num_hidden_layers, override) rather than a bare override: num_hidden_layers
is rewritten from the checkpoint's config.json by update_model_arch, so a
variant with fewer layers must clamp rather than ask for layers that do not exist.

The tapped entry is bit-identical

The hidden-state tuple appends before each layer runs, so entry N is the input
to layer N, which is the output of layer N-1:

for layer_index, layer in enumerate(self.layers):
    if all_hidden_states is not None:
        all_hidden_states += (hidden_states, )
    hidden_states = layer(hidden_states, position_embeddings, attention_mask)

Stopping after N layers and appending the running tensor once more puts the same
value at the same index.

The final norm has to go with the layers

This is the part that fails quietly if you get it wrong. self.norm sits above
the tap. A truncated stack that still applied it would append a normalised
tensor at index 50, the length check at minimax_h3_conditioning.py:181 would
still pass, and conditioning would change with nothing raised. So self.norm is
None when truncated, which is also what load_weights keys off to drop the
surplus checkpoint entries. The unexpected-key check there is strict on purpose,
so the surplus is filtered rather than the check relaxed.

Test Plan

pytest fastvideo/tests/encoders/test_minimax_h3_qwen3_vl_truncation.py -q
pre-commit run --files fastvideo/configs/models/encoders/minimax_h3_qwen3_vl.py \
                       fastvideo/models/encoders/minimax_h3_qwen3_vl.py \
                       fastvideo/tests/encoders/test_minimax_h3_qwen3_vl_truncation.py

CPU only. Covered: the default equals the constant the pipeline reads, which the
two sides cannot import from each other; the stack builds to the override and
drops the norm; None keeps the full stack; an override above the stack clamps;
the tapped hidden state is torch.equal across truncated and full models given
the same weights; the surplus checkpoint keys are filtered while the built
layers, embeddings and vision tower are not.

Test Results

Test output

$ pytest fastvideo/tests/encoders/test_minimax_h3_qwen3_vl_truncation.py -q
.......
7 passed, 14 warnings in 3.21s

$ pre-commit run --files ...
yapf.....................................................................Passed
ruff (legacy alias)......................................................Passed
codespell................................................................Passed

mypy could not run in my checkout, whose directory name contains a hyphen and so
is not a valid package name. It passes in CI.

Measured on a DGX Spark (GB10, 121 GB unified memory), loading
noctuashap/MiniMax-H3-pruned-r16, memory sampled every 5 s with free, both
runs stacked on #1710 so the offload path is out of the way:

	resident after Loaded module text_encoder
without this change	67 GB
with this change	52 GB

15 GB against the 13.7 GB the arithmetic predicts; the difference is allocator
granularity. The real checkpoint loads without raising on the surplus keys,
which is the part the unit tests cannot cover.

Not verified: a completed generation. That device cannot load H3 at all yet.
With #1710 and this change it now gets through the text encoder, tokenizer,
processor, VAE and audio VAE and dies partway into the transformer, at 113 GB of
121 with 26 GB already taken by another user. Extrapolating from the component
sizes, an idle machine needs about 109 GB, so it should complete; I have not yet
had the machine to myself to show it. Happy to post that when I do.

Scope

Training reads precomputed embeddings from parquet and never runs the encoder.
Preprocessing runs it but reads the same index. FL2VA and Ref2VA share the same
conditioning stage and the same index, so they are consistent rather than
broken. Setting the override to None restores the full stack for anyone who
wants the top of the tower, at the cost of the 13.7 GB.

Load time improves less than memory does: weight_utils still reads every
tensor off disk, only the copy into a parameter disappears.

Checklist

• I ran pre-commit on the changed files and fixed all issues
• I added or updated tests for my changes
• I updated documentation if needed
• I considered GPU memory impact of my changes

[mergify]

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

• #approved-reviews-by>=1
• check-success=fastcheck-passed
• check-success=full-suite-passed

This rule is failing.

• #approved-reviews-by>=1
• check-success=fastcheck-passed
• check-success=full-suite-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ec849a4e3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Update the production-loader parity test for truncation

When MINIMAX_H3_RUN_ENCODER_PARITY=1, the existing tests/local_tests/encoders/test_minimax_h3_qwen3_vl_parity.py test now deterministically fails: the official 64-layer model returns 65 hidden states, while this default truncation returns 51, but lines 224–226 still require equal tuple lengths and compare every layer with zip(..., strict=True). Adjust that required production-loader parity check to compare the shared prefix or specifically layer 50; otherwise the real-checkpoint numerical gate cannot validate this optimization.

Useful? React with 👍 / 👎.

[SolitaryThinker]

Reviewed in depth — the tap math is right (hidden_states records each
layer's input, so entry 50 is the raw output of layer 49 in both stacks, and
dropping the final norm when truncated is exactly what keeps it
bit-identical), override=None is byte-identical to today, preprocess taps
the same constant through the same stage, and training never builds this
encoder. I pushed a follow-up fix commit (a02634fc7) directly onto
truncate-h3-text-encoder via maintainer edit (also mirrored as
pr1711-fixes in this repo):

1. test_minimax_h3_qwen3_vl_parity.py (the real-checkpoint gate flagged by
   Codex) now pins the production tuple to num_layers + 1 entries and
   compares every built hidden state bit-exactly against the official model
   at the same index — still catches a truncated stack that applies the
   final norm, and is the unchanged full comparison when the override is None.
2. num_hidden_layers_override <= 0 is now rejected in __post_init__
   (construction + update_model_arch): 0 built an empty stack, and a
   negative value made the surplus-key filter drop every layer key so the
   conditioner "loaded" with no transformer at all.
3. _is_above_the_tap is bounded to [num_layers, num_hidden_layers) so
   truncation only drops keys the full stack would have accepted; corrupt
   above-config keys still raise.
4. The CPU truncation test now pins the whole shared prefix (plus the
   override == num_hidden_layers boundary), mirroring the parity gate.

9/9 truncation tests pass on CPU; pre-commit (incl. mypy) passes on the five
touched files. One ask before merge: run the encoder parity gate once on a
GPU node with the real checkpoint
(MINIMAX_H3_RUN_ENCODER_PARITY=1 pytest tests/local_tests/encoders/test_minimax_h3_qwen3_vl_parity.py -v -s).
Serving note: with 13.7 GB back, configs can disable
text_encoder_cpu_offload and keep the conditioner resident.

[SolitaryThinker]

Ran the GPU gate for this PR at head a02634fc7 on a GB200 (driver 580.82.07, torch 2.12.0+cu130, transformers 5.15.0) with the real MiniMax-H3 checkpoint:

• Truncation unit tests: 9/9 pass on GPU.
• Encoder parity gate, text case: PASS — every built hidden state (51 entries) bit-exact vs the official full stack at atol=0; tapped-entry drift max_abs=0.00000000 mean_abs=0.00000000.
• E2E conditioning smoke (production TextEncoderLoader + MiniMaxH3ConditioningStage, 2 real prompts): override=50 vs override=None embeddings are bit-identical (max_abs_diff=0, shapes (1,514,5120)/(1,303,5120) bf16).
• Memory claim confirmed: 47.98 GiB vs 60.70 GiB weights resident = 13.65 GB saved (params 25.753B vs 32.579B, Δ6.826B ≈ the stated 6.83B/13.7 GB). Load time is unchanged (~7.5 s warm; the loader still scans all 14 shards).
• The gate's image/video cases fail, but this is pre-existing and unrelated to this PR: the divergence is at hidden_states[0] (the embeddings entry, below any decoder layer), confined exactly to <|image_pad|>/<|video_pad|> positions, and reproduces byte-for-byte at the PR base 8208536cd with the full 64-layer stack (image layer 0: 338336/445440 (76.0%) mismatched, max abs 1.125 — identical at base and head). That is a FastVideo-vision-tower vs transformers-5.15.0 parity issue on main, worth tracking on its own. All text positions are bit-identical everywhere.

Net: everything this PR touches is bit-exact on the real checkpoint; the memory saving is as advertised.

🤖 Generated with Claude Code