Skip to content

[perf] Disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 - #1715

Merged
SolitaryThinker merged 2 commits into
hao-ai-lab:mainfrom
KyleNeverGivesUp:skip-dit-offload-on-unified-memory
Aug 26, 2026
Merged

[perf] Disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10#1715
SolitaryThinker merged 2 commits into
hao-ai-lab:mainfrom
KyleNeverGivesUp:skip-dit-offload-on-unified-memory

Conversation

@KyleNeverGivesUp

@KyleNeverGivesUp KyleNeverGivesUp commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Purpose

CPU offload reduces accelerator pressure only when host and accelerator memory
are separate physical pools. On GB10, Jetson-class integrated CUDA devices, and
Apple silicon, moving a component to the host does not free the underlying pool
and can increase peak memory through transient copies.

This PR applies one worker-local unified-memory policy to every component
offload mode while preserving explicit FSDP inference sharding.

Changes

  • Classify the selected device after each worker binds its local rank; the
    driver does not initialize CUDA or classify every worker from device zero.
  • Clear the five host-placement modes on unified memory:
    dit_cpu_offload, dit_layerwise_offload,
    text_encoder_cpu_offload, image_encoder_cpu_offload, and
    vae_cpu_offload.
  • Defer inference conflict normalization until device classification. Unified
    CUDA therefore clears layerwise offload before resolving the conflict and
    preserves an explicit use_fsdp_inference=True; discrete CUDA retains the
    historical layerwise-offload precedence.
  • Cache classification per arguments object and device ID while still clearing
    any offload flag that a later caller re-enables.
  • Apply the same policy to direct pipeline construction and role-aware encoder
    loading, including explicit CPU targets.
  • Keep pin_cpu_memory outside the policy because it also controls output
    transfers rather than component placement.
  • Collect the CPU-safe platform and pipeline tests in the unit lane and document
    the automatic policy in the offloading guide.

Dependency

This PR is rebased onto current main after merged #1710 and #1714. Its final
branch contains only the unified-memory extension and maintainer fix after the
reviewed loader/worker foundation. Final head:
6fb6af9fabb633b235be67e81288da1558fd81f1.

Verification

focused policy, loader, pipeline, and worker tests: 45 passed
post-#1714 loader/platform integration checks: 13 passed
pinned MiniMax-H3 packing/media/scheduler and pipeline checks: 37 passed
related LingBot/Z-Image coverage: 18 passed, 4 explicit opt-in skips
exact unit lane: 988 passed, 7 skipped
pre-commit and diff checks: passed

A real NVIDIA GB10 policy probe started from production defaults with explicit
FSDP inference enabled. It classified the device as unified memory, cleared all
five host-offload flags, and preserved FSDP. The combined #1710/#1714/#1715
stack was also exercised by the later full FastH3 lifecycle run used to validate
#1761.

Checklist

  • Tests cover each flag, worker-local classification, caching, and FSDP
  • Applicable pre-commit hooks pass
  • General offloading documentation is updated
  • Memory behavior was validated on GB10

@mergify mergify Bot added type: perf Performance improvement scope: infra CI, tests, Docker, build scope: model Model architecture (DiTs, encoders, VAEs) labels Aug 18, 2026
@mergify

mergify Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

Protection Waiting on
🔴 PR merge requirements 👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

  • #approved-reviews-by>=1
  • check-success=fastcheck-passed
  • check-success=full-suite-passed
This rule is failing.
  • #approved-reviews-by>=1
  • check-success=fastcheck-passed
  • check-success=full-suite-passed
  • check-success~=pre-commit
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5e06f62897

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread fastvideo/platforms/cuda.py Outdated
Comment thread fastvideo/fastvideo_args.py Outdated
@KyleNeverGivesUp KyleNeverGivesUp changed the title [perf]: disable every offload path on unified memory, not just the loader's disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 Aug 18, 2026
@mergify

mergify Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

⚠️ PR title format required

Your PR title must start with a type tag in brackets. Examples:

  • [feat] Add new model support
  • [bugfix] Fix VAE tiling corruption
  • [refactor] Restructure training pipeline
  • [perf] Optimize attention kernel
  • [ci] Update test infrastructure
  • [infra] Add activation trace hooks
  • [docs] Add inference guide
  • [misc] Clean up configs
  • [new-model] Port Flux2 to FastVideo
  • [skill] Add add-model agent skill

Valid tags: feat, feature, bugfix, fix, refactor, perf, ci, infra, doc, docs, misc, chore, kernel, new-model, skill, skills

Please update your PR title and the merge protection check will pass automatically.

@KyleNeverGivesUp KyleNeverGivesUp changed the title disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 [perf]: disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 Aug 18, 2026
@KyleNeverGivesUp KyleNeverGivesUp changed the title [perf]: disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 [perf]: Disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 Aug 18, 2026
@KyleNeverGivesUp KyleNeverGivesUp changed the title [perf]: Disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 [perf] Disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 Aug 18, 2026
@SolitaryThinker
SolitaryThinker force-pushed the skip-dit-offload-on-unified-memory branch from 5e06f62 to de12e08 Compare August 23, 2026 22:55
@mergify mergify Bot added the scope: inference Inference pipeline, serving, CLI label Aug 23, 2026
@SolitaryThinker
SolitaryThinker force-pushed the skip-dit-offload-on-unified-memory branch from de12e08 to 028e3fe Compare August 23, 2026 23:38
@KyleNeverGivesUp
KyleNeverGivesUp force-pushed the skip-dit-offload-on-unified-memory branch from 028e3fe to 1cee9ba Compare August 26, 2026 10:29
@SolitaryThinker
SolitaryThinker force-pushed the skip-dit-offload-on-unified-memory branch from 1cee9ba to 20cbc36 Compare August 26, 2026 19:13
@SolitaryThinker

Copy link
Copy Markdown
Collaborator

/merge

@github-actions github-actions Bot added the ready PR is ready to merge label Aug 26, 2026
@mergify mergify Bot added the scope: docs Documentation label Aug 26, 2026
@SolitaryThinker

Copy link
Copy Markdown
Collaborator

/test full

KyleNeverGivesUp and others added 2 commits August 26, 2026 15:25
CPU offload only saves memory when host RAM and accelerator memory are separate pools. On GB10, Jetson, and Apple silicon the move frees nothing and can retain two copies at peak, so all five component offload flags must share one unified-memory decision.

Make that decision inside each worker after it binds its logical device, before pipeline construction. This avoids initializing CUDA in the parent process and lets heterogeneous workers classify their own devices instead of inheriting a device-0 result. Direct build_pipeline and from_pretrained callers apply the same policy after distributed setup has selected their device.

The centralized flag inventory covers DiT full and layerwise offload, text and image encoders, and VAE offload. Role-aware encoder loading also resets explicit CPU targets, while use_fsdp_inference remains unchanged because sharding is orthogonal to host placement. Tests cover exact flag inventory, non-vacuous per-flag behavior, discrete CUDA, MPS, per-device divergence, direct pipeline ordering, and inherited image-loader placement.

On a DGX Spark this removes the 48 GB conditioning-time encoder shuttle and allows MiniMax H3 to complete conditioning and denoising. Video decode precision remains a separate concern because only the decode half of the VAE can safely use lower precision.
@SolitaryThinker
SolitaryThinker force-pushed the skip-dit-offload-on-unified-memory branch from 20cbc36 to 6fb6af9 Compare August 26, 2026 22:26
@SolitaryThinker

Copy link
Copy Markdown
Collaborator

/merge

@SolitaryThinker

Copy link
Copy Markdown
Collaborator

/test full

@github-actions github-actions Bot added ready PR is ready to merge and removed ready PR is ready to merge labels Aug 26, 2026
@SolitaryThinker

Copy link
Copy Markdown
Collaborator

/test fastcheck

@SolitaryThinker
SolitaryThinker merged commit e9bbaca into hao-ai-lab:main Aug 26, 2026
2 of 6 checks passed
KyleNeverGivesUp added a commit to KyleNeverGivesUp/FastVideo that referenced this pull request Aug 27, 2026
Both sides added a bullet to the GB10 tuning list and git could not tell
they were about different things. hao-ai-lab#1715 documents that FastVideo now
disables the offload modes once a worker binds its device; this branch
documents that earlyoom prefers Python and that a worker's SIGTERM
traceback shows where it was interrupted, not why it was chosen. Neither
replaces the other, so both are kept, offload first.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready PR is ready to merge scope: docs Documentation scope: inference Inference pipeline, serving, CLI scope: infra CI, tests, Docker, build scope: model Model architecture (DiTs, encoders, VAEs) type: perf Performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants