[perf] Disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 - #1715
Conversation
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI
🔴 PR merge requirementsWaiting for
This rule is failing.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e06f62897
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
5e06f62 to
de12e08
Compare
de12e08 to
028e3fe
Compare
028e3fe to
1cee9ba
Compare
1cee9ba to
20cbc36
Compare
|
/merge |
|
/test full |
CPU offload only saves memory when host RAM and accelerator memory are separate pools. On GB10, Jetson, and Apple silicon the move frees nothing and can retain two copies at peak, so all five component offload flags must share one unified-memory decision. Make that decision inside each worker after it binds its logical device, before pipeline construction. This avoids initializing CUDA in the parent process and lets heterogeneous workers classify their own devices instead of inheriting a device-0 result. Direct build_pipeline and from_pretrained callers apply the same policy after distributed setup has selected their device. The centralized flag inventory covers DiT full and layerwise offload, text and image encoders, and VAE offload. Role-aware encoder loading also resets explicit CPU targets, while use_fsdp_inference remains unchanged because sharding is orthogonal to host placement. Tests cover exact flag inventory, non-vacuous per-flag behavior, discrete CUDA, MPS, per-device divergence, direct pipeline ordering, and inherited image-loader placement. On a DGX Spark this removes the 48 GB conditioning-time encoder shuttle and allows MiniMax H3 to complete conditioning and denoising. Video decode precision remains a separate concern because only the decode half of the VAE can safely use lower precision.
20cbc36 to
6fb6af9
Compare
|
/merge |
|
/test full |
|
/test fastcheck |
Both sides added a bullet to the GB10 tuning list and git could not tell they were about different things. hao-ai-lab#1715 documents that FastVideo now disables the offload modes once a worker binds its device; this branch documents that earlyoom prefers Python and that a worker's SIGTERM traceback shows where it was interrupted, not why it was chosen. Neither replaces the other, so both are kept, offload first.
Purpose
CPU offload reduces accelerator pressure only when host and accelerator memory
are separate physical pools. On GB10, Jetson-class integrated CUDA devices, and
Apple silicon, moving a component to the host does not free the underlying pool
and can increase peak memory through transient copies.
This PR applies one worker-local unified-memory policy to every component
offload mode while preserving explicit FSDP inference sharding.
Changes
driver does not initialize CUDA or classify every worker from device zero.
dit_cpu_offload,dit_layerwise_offload,text_encoder_cpu_offload,image_encoder_cpu_offload, andvae_cpu_offload.CUDA therefore clears layerwise offload before resolving the conflict and
preserves an explicit
use_fsdp_inference=True; discrete CUDA retains thehistorical layerwise-offload precedence.
any offload flag that a later caller re-enables.
loading, including explicit CPU targets.
pin_cpu_memoryoutside the policy because it also controls outputtransfers rather than component placement.
the automatic policy in the offloading guide.
Dependency
This PR is rebased onto current
mainafter merged #1710 and #1714. Its finalbranch contains only the unified-memory extension and maintainer fix after the
reviewed loader/worker foundation. Final head:
6fb6af9fabb633b235be67e81288da1558fd81f1.Verification
A real NVIDIA GB10 policy probe started from production defaults with explicit
FSDP inference enabled. It classified the device as unified memory, cleared all
five host-offload flags, and preserved FSDP. The combined #1710/#1714/#1715
stack was also exercised by the later full FastH3 lifecycle run used to validate
#1761.
Checklist