Repository navigation
[Feature] Add Dreamverse mode selection for T2VA, FL2VA, and Ref2VA #1834
Copy link
Copy link
Open
Labels
installationInstallation and setup issuesInstallation and setup issuesplatformPlatform-specific (Windows/macOS)Platform-specific (Windows/macOS)scope: attentionAttention backends (VSA, STA, Flash, etc.)Attention backends (VSA, STA, Flash, etc.)scope: docsDocumentationDocumentationscope: inferenceInference pipeline, serving, CLIInference pipeline, serving, CLIscope: modelModel architecture (DiTs, encoders, VAEs)Model architecture (DiTs, encoders, VAEs)
Description
Motivation
Dreamverse currently starts every project through one implicit generation path. With FastH3 support now available, users need to declare whether a project is text-only, keyframe-conditioned, or driven by ordered multimodal references before generation begins.
This issue tracks the end-to-end mode-selection experience for the three MiniMax H3 task families:
- T2VA — text to video and audio.
- FL2VA — first/last-frame to video and audio.
- Ref2VA — ordered image, video, and audio references to video and audio.
Use these canonical identifiers in code and protocol payloads: t2va, fl2va, and ref2va.
Proposed scope
Frontend
- Add an accessible generation-mode selector to the pre-session Dreamverse composer.
- Default new projects to T2VA.
- Explain the input requirements for each mode.
- Keep the selected mode stable for a project and lock mode changes once its generation session starts.
- Show mode-specific inputs:
- T2VA: prompt only.
- FL2VA: first frame and optional last frame, subject to the final product decision.
- Ref2VA: ordered image/video/audio references.
- Integrate selected assets from the planned Asset List /
@picker without duplicating asset ownership in the composer.
Protocol and runtime
- Include
generation_modeand mode-specific conditioning inputs in project/session initialization. - Validate mode names and conditioning requirements at the runtime boundary with actionable errors.
- Route T2VA/FL2VA requests to the H3 base pipeline and Ref2VA requests to the H3 reference pipeline/profile.
- Preserve the current WebSocket/session/GPU-pool ownership boundaries.
Tests
- Cover canonical mode metadata and the T2VA default.
- Cover selection, disabled/busy behavior, and project-init serialization.
- Cover runtime validation for valid modes, unknown modes, and missing/incompatible conditioning inputs.
- Add a mock-server path so the UI flow can be tested without GPUs.
Acceptance criteria
- A user can choose T2VA, FL2VA, or Ref2VA before starting a project.
- The UI clearly communicates and collects inputs required by the selected mode.
- The selected mode and ordered conditioning assets reach the runtime through a typed payload.
- The runtime rejects invalid combinations and dispatches valid requests to the correct H3 path.
- Existing LTX/Dreamverse flows remain backward-compatible when
generation_modeis absent. - Frontend, protocol, mock-server, and runtime tests pass.
Dependencies and coordination
- FastH3 Dreamverse backend foundation: Add H3 support into Dreamverse #1800 (merged).
- Asset List is tracked as Aryan's work in the Dreamverse V2 plan; this issue should consume that surface for FL2VA/Ref2VA attachment selection.
- Confirm whether FL2VA accepts first-only and last-only inputs in the product UI, or requires both endpoints.
- Confirm model/profile routing for FastH3 Preview versus base H3 checkpoints before enabling real FL2VA/Ref2VA generation.
Related resources
- Dreamverse V2 planning document: https://docs.google.com/document/d/1Dzty4rIXVNUXZDZH47ia6aw6Ad88RhgBcEMwJ43f2Fs/edit
- FastH3 Dreamverse integration: Add H3 support into Dreamverse #1800
- MiniMax H3: https://github.com/MiniMax-AI/MiniMax-H3
Metadata
Metadata
Assignees
Labels
installationInstallation and setup issuesInstallation and setup issuesplatformPlatform-specific (Windows/macOS)Platform-specific (Windows/macOS)scope: attentionAttention backends (VSA, STA, Flash, etc.)Attention backends (VSA, STA, Flash, etc.)scope: docsDocumentationDocumentationscope: inferenceInference pipeline, serving, CLIInference pipeline, serving, CLIscope: modelModel architecture (DiTs, encoders, VAEs)Model architecture (DiTs, encoders, VAEs)
Activity
github-actions commentedon Sep 9, 2026
Welcome to FastVideo! Thanks for opening your first issue.
To help us investigate, please include:
pip show fastvideonvidia-smioutput (GPU model, driver, CUDA version)python --versionIf this is a bug, a minimal reproduction script helps us fix it faster.
Useful links:
[feat]: align backend wiring with upstream generation_mode contract