[new-model] Add Cosmos Predict2.5 distilled T2W and DFD V2W inference by Mister-Raggs · Pull Request #1767 · hao-ai-lab/FastVideo

Mister-Raggs · 2026-08-27T20:34:08Z

Purpose

Add FastVideo inference support for two complementary public Cosmos Predict2.5 2B students:

NVIDIA's distilled Text2World student for four-step bootstrap generation.
The Data-Forcing Distillation (DFD) Video2World student for four-step, one-frame-conditioned continuation.

The implementations preserve each release's inference equations and checkpoint layout instead of routing either student through the full model's UniPC path. Existing Cosmos Predict2.5 inference remains unchanged.

Changes

Add Cosmos25DistilledScheduler for the official fixed-noise TrigFlow rollout.
Add Cosmos25DFDScheduler for the released DFD rectified-flow ODE schedule: [0.999, 0.937, 0.833, 0.624, 0.0].
Select isolated scheduler-specific T2W and DFD V2W pipeline stages from packaged metadata.
Implement DFD's one-frame image latent, condition mask, clean-frame replacement, per-frame timestep, and 24 FPS RoPE semantics.
Add converters for NVIDIA's monolithic distilled checkpoint and DFD's sharded PyTorch DCP checkpoint. Both converters retain only inference student tensors and reuse the existing Cosmos tokenizer, VAE, Reason1 encoder, and safety components.
Add runnable T2W and DFD I2W examples, CPU contracts, pinned upstream scheduler tests, real-weight DiT comparisons, full four-step rollout parity, and validation notes.
Fall back from unsupported Qwen2.5-VL eager vision attention to Torch SDPA for the Reason1 encoder.

Media

Distilled T2W output

cosmos25_distilled_full.mp4

Native FastVideo DFD output

fastvideo_dfd_i2w.mp4

Test plan and results

Distilled T2W

Scheduler unit tests: 7 passed.
Pinned official scheduler parity: 2 passed, non-skipped.
Conversion contracts: 7 passed.
Pipeline contracts: 9 passed.
Released checkpoint converted to 685 student tensors with no training counters; production strict load passed.
Official-vs-FastVideo BF16 DiT parity passed: first-block relative mean 0.000655, final relative mean 0.038397, final mean absolute 0.037414.
Small one-step 256x448x9 generation and decoded-frame return passed.
Full four-step 704x1280x77 generation passed in 143.53 seconds after load and passed visual inspection.

DFD V2W

Scheduler unit tests: 10 passed.
Pinned upstream scheduler parity: 1 passed, non-skipped.
Conversion contracts: 4 passed.
Real-weight BF16 DiT parity passed: maximum absolute 0.15625, mean absolute 0.01315392, relative mean 0.01986194.
Full real-weight four-step latent rollout matched upstream with identical image latent, text embeddings, and initial noise.
The public DCP checkpoint converted to 569 student tensors / 2,059,174,912 parameters and strict-loaded through the production FastVideo loader.
Native four-step 704x1280x81 generation passed: 145.20 seconds generation, 147.66 seconds end to end.
Native boundary check: input to frame 0 MAE 2.845237; input to frame 80 control 52.130680.
Hybrid model evidence passed across T2W -> DFD and DFD -> DFD boundaries: MAE 2.694 and 2.459, versus unrelated control 39.142; both boundaries were visually seamless and the prompt-directed right pivot succeeded.

Static and regression checks

Relevant scheduler, converter, pipeline, preset, encoder, and parity suites pass in the FastVideo validation environment.
YAPF, Ruff, codespell, PyMarkdown, and CI mypy pass.
git diff --check passes.
Existing full-step Cosmos Predict2.5 routing remains covered and unchanged.

Runtime contracts and limitations

Distilled T2W is the unconditioned bootstrap path; the released DFD checkpoint requires an input image.
DFD accepts exactly one conditioning image and produces 81 frames at 704x1280, 24 FPS, four steps, and guidance scale 1.
The first decoded DFD frame repeats the conditioning image by design. Continuation consumers should trim that frame at segment boundaries.
These students do not generate audio.
This is not real-time on GB10: a full segment takes roughly 145 seconds after model load.
Users must run the included converters and provide local package directories until public converted FastVideo package IDs are published.
No SSIM baseline exists yet; both full outputs received manual visual quality review in addition to numerical parity and boundary checks.
This branch still carries the two Reason1 fixes from [bugfix] reason1: handle BatchEncoding from apply_chat_template #1607 so the validated path is reproducible; they can be dropped if [bugfix] reason1: handle BatchEncoding from apply_chat_template #1607 lands first.

Checklist

Existing Cosmos Predict2.5 paths remain unchanged.
Distilled and DFD schedulers match their pinned upstream references.
Real checkpoints convert and strict-load without training-only state.
Real-weight DiT and complete DFD rollout parity pass.
Full-resolution T2W and DFD outputs pass runtime and visual gates.
Relevant pre-commit hooks pass.
Attach the native DFD MP4 under Media.
Publish and register converted T2W and DFD package IDs.

mergify · 2026-08-27T20:34:50Z

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

#approved-reviews-by>=1
check-success=fastcheck-passed
check-success=full-suite-passed

This rule is failing.

#approved-reviews-by>=1
check-success=fastcheck-passed
check-success=full-suite-passed
check-success~=pre-commit
title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

mergify · 2026-09-07T23:15:53Z

This PR has merge conflicts with the base branch. Please rebase:

git fetch origin main
git rebase origin/main
# Resolve any conflicts, then:
git push --force-with-lease

Cosmos-Predict2.5 2B text encoding crashes on load because `apply_chat_template(tokenize=True)` returns a transformers `BatchEncoding` -- a `collections.UserDict`, i.e. a `Mapping` but NOT a `dict` -- which the `isinstance(tokenizer_output, dict)` check missed, so the id extraction fell through to `raise RuntimeError("Unexpected chat_template output type")`. Match on `Mapping` instead (every `dict` is a `Mapping`, so existing plain-dict and list outputs are unchanged) and lift the normalization into a tested `_normalize_chat_template_ids` helper. Add weight-free parametrized regression tests over every shape apply_chat_template can return.

Address review: the helper documents a flat list[int] contract, but a nested list with batch>1 slipped through the len==1 unwrap guard and was returned as-is, violating the contract and breaking downstream padding. Raise on batch>1 instead of silently returning a nested list; add tests.

mergify · 2026-09-11T07:14:15Z

Pre-commit checks failed

Hi @Mister-Raggs, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

yapf: yapf -i <file> (formatting)
ruff: ruff check --fix <file> (linting)
codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

mergify · 2026-09-11T07:19:28Z

Pre-commit checks failed

Hi @Mister-Raggs, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

yapf: yapf -i <file> (formatting)
ruff: ruff check --fix <file> (linting)
codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

mergify · 2026-09-11T07:23:58Z

Pre-commit checks failed

Hi @Mister-Raggs, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

yapf: yapf -i <file> (formatting)
ruff: ruff check --fix <file> (linting)
codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

mergify · 2026-09-11T07:30:32Z

Pre-commit checks failed

Hi @Mister-Raggs, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

yapf: yapf -i <file> (formatting)
ruff: ruff check --fix <file> (linting)
codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

mergify Bot added type: new-model New model support scope: inference Inference pipeline, serving, CLI scope: infra CI, tests, Docker, build scope: docs Documentation scope: model Model architecture (DiTs, encoders, VAEs) labels Aug 27, 2026

Mister-Raggs mentioned this pull request Aug 27, 2026

[feat] Add Cosmos Predict2.5 DFD continuation to DreamVerse #1768

Open

7 tasks

Mister-Raggs changed the title ~~[New Model] Add Cosmos Predict2.5 2B distilled T2W inference~~ [new-model] Add Cosmos Predict2.5 2B distilled T2W inference Aug 27, 2026

Mister-Raggs marked this pull request as ready for review August 27, 2026 20:54

Copilot AI lite review requested due to automatic review settings August 27, 2026 20:54

Copilot AI reviewed Aug 27, 2026

mergify Bot added the needs-rebase PR has merge conflicts label Sep 7, 2026

Mister-Raggs added 17 commits September 10, 2026 23:13

[feat] add Cosmos 2.5 distilled sampler scaffold

3f8e23c

[fix] exclude Cosmos training counters from conversion

e47ac68

[test] add Cosmos 2.5 distilled DiT parity gate

e1d3bd2

[test] bypass Cosmos package CUDA-extra guard

db0317f

test: isolate Cosmos DiT parity from config deps

edab5c6

test: run Cosmos reference DiT without transformer engine

d308c65

test: complete reference RMSNorm interface

210e869

test: ignore Cosmos reference training counters

a3b7d4a

test: trace Cosmos distilled DiT parity drift

5c5a112

test: calibrate Cosmos BF16 parity tolerance

03ce5c1

[Model] wire Cosmos2.5 distilled T2W inference

7e5e9e5

[Test] match Cosmos distilled mask dtype

bc3d19e

[Test] add Cosmos distilled frame return gate

2ca07cb

[Docs] record Cosmos 2.5 distilled validation

b824bb6

[Model] add Cosmos2.5 DFD component scaffold

69da225

Mister-Raggs force-pushed the feat/cosmos25-distilled-sampler branch from 72f25e4 to 69da225 Compare September 11, 2026 06:39

Mister-Raggs added 2 commits September 10, 2026 23:48

[fix] match DFD RF precision boundaries

773fb02

[test] make Cosmos parity rendezvous self-contained

74d18f5

mergify Bot removed the needs-rebase PR has merge conflicts label Sep 11, 2026

Mister-Raggs added 4 commits September 10, 2026 23:53

[test] calibrate DFD BF16 parity bounds

7b915ab

[Model] wire Cosmos2.5 DFD V2W pipeline

b5ba654

[test] provide valid DFD preset manifest

f0e1da4

[Docs] record Cosmos2.5 DFD pipeline parity

ab7fa99

Mister-Raggs added 2 commits September 11, 2026 00:15

[bugfix] fall back Reason1 eager attention to SDPA

9027f02

[test] complete tiny Qwen vision config

dda0f9e

[Docs] record DFD production smoke

f3deeb5

[Docs] accept DFD production quality gate

85b3060

[misc] apply YAPF to DFD pipeline imports

652307f

Mister-Raggs changed the title ~~[new-model] Add Cosmos Predict2.5 2B distilled T2W inference~~ [new-model] Add Cosmos Predict2.5 distilled T2W and DFD V2W inference Sep 11, 2026

Navigation Menu

Uh oh!

[new-model] Add Cosmos Predict2.5 distilled T2W and DFD V2W inference - #1767

[new-model] Add Cosmos Predict2.5 distilled T2W and DFD V2W inference#1767
Mister-Raggs wants to merge 28 commits into
hao-ai-lab:mainfrom
Mister-Raggs:feat/cosmos25-distilled-sampler

Mister-Raggs commented Aug 27, 2026 •

edited

Loading

Uh oh!

mergify Bot commented Aug 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

mergify Bot commented Sep 7, 2026

Uh oh!

mergify Bot commented Sep 11, 2026

Uh oh!

mergify Bot commented Sep 11, 2026

Uh oh!

mergify Bot commented Sep 11, 2026

Uh oh!

mergify Bot commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Mister-Raggs commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Changes

Media

Distilled T2W output

Native FastVideo DFD output

Test plan and results

Distilled T2W

DFD V2W

Static and regression checks

Runtime contracts and limitations

Checklist

Uh oh!

mergify Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🔴 PR merge requirements

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Sep 7, 2026

Uh oh!

mergify Bot commented Sep 11, 2026

Pre-commit checks failed

Uh oh!

mergify Bot commented Sep 11, 2026

Pre-commit checks failed

Uh oh!

mergify Bot commented Sep 11, 2026

Pre-commit checks failed

Uh oh!

mergify Bot commented Sep 11, 2026

Pre-commit checks failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Mister-Raggs commented Aug 27, 2026 •

edited

Loading

mergify Bot commented Aug 27, 2026 •

edited

Loading