Skip to content

[Feature] Add Dreamverse mode selection for T2VA, FL2VA, and Ref2VA #1834

Open

Description

@jayzou3773

Motivation

Dreamverse currently starts every project through one implicit generation path. With FastH3 support now available, users need to declare whether a project is text-only, keyframe-conditioned, or driven by ordered multimodal references before generation begins.

This issue tracks the end-to-end mode-selection experience for the three MiniMax H3 task families:

  • T2VA — text to video and audio.
  • FL2VA — first/last-frame to video and audio.
  • Ref2VA — ordered image, video, and audio references to video and audio.

Use these canonical identifiers in code and protocol payloads: t2va, fl2va, and ref2va.

Proposed scope

Frontend

  • Add an accessible generation-mode selector to the pre-session Dreamverse composer.
  • Default new projects to T2VA.
  • Explain the input requirements for each mode.
  • Keep the selected mode stable for a project and lock mode changes once its generation session starts.
  • Show mode-specific inputs:
    • T2VA: prompt only.
    • FL2VA: first frame and optional last frame, subject to the final product decision.
    • Ref2VA: ordered image/video/audio references.
  • Integrate selected assets from the planned Asset List / @ picker without duplicating asset ownership in the composer.

Protocol and runtime

  • Include generation_mode and mode-specific conditioning inputs in project/session initialization.
  • Validate mode names and conditioning requirements at the runtime boundary with actionable errors.
  • Route T2VA/FL2VA requests to the H3 base pipeline and Ref2VA requests to the H3 reference pipeline/profile.
  • Preserve the current WebSocket/session/GPU-pool ownership boundaries.

Tests

  • Cover canonical mode metadata and the T2VA default.
  • Cover selection, disabled/busy behavior, and project-init serialization.
  • Cover runtime validation for valid modes, unknown modes, and missing/incompatible conditioning inputs.
  • Add a mock-server path so the UI flow can be tested without GPUs.

Acceptance criteria

  • A user can choose T2VA, FL2VA, or Ref2VA before starting a project.
  • The UI clearly communicates and collects inputs required by the selected mode.
  • The selected mode and ordered conditioning assets reach the runtime through a typed payload.
  • The runtime rejects invalid combinations and dispatches valid requests to the correct H3 path.
  • Existing LTX/Dreamverse flows remain backward-compatible when generation_mode is absent.
  • Frontend, protocol, mock-server, and runtime tests pass.

Dependencies and coordination

  • FastH3 Dreamverse backend foundation: Add H3 support into Dreamverse #1800 (merged).
  • Asset List is tracked as Aryan's work in the Dreamverse V2 plan; this issue should consume that surface for FL2VA/Ref2VA attachment selection.
  • Confirm whether FL2VA accepts first-only and last-only inputs in the product UI, or requires both endpoints.
  • Confirm model/profile routing for FastH3 Preview versus base H3 checkpoints before enabling real FL2VA/Ref2VA generation.

Related resources

Activity

  1. added
    installationInstallation and setup issues
    platformPlatform-specific (Windows/macOS)
    scope: attentionAttention backends (VSA, STA, Flash, etc.)
    scope: inferenceInference pipeline, serving, CLI
    scope: modelModel architecture (DiTs, encoders, VAEs)
    on Sep 9, 2026
  2. github-actions commented on Sep 9, 2026

    @github-actions

    Welcome to FastVideo! Thanks for opening your first issue.

    To help us investigate, please include:

    • FastVideo version: pip show fastvideo
    • GPU: nvidia-smi output (GPU model, driver, CUDA version)
    • Python version: python --version
    • OS: e.g., Ubuntu 22.04

    If this is a bug, a minimal reproduction script helps us fix it faster.

    Useful links:

  3. self-assigned this
    on Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

installationInstallation and setup issuesplatformPlatform-specific (Windows/macOS)scope: attentionAttention backends (VSA, STA, Flash, etc.)scope: docsDocumentationscope: inferenceInference pipeline, serving, CLIscope: modelModel architecture (DiTs, encoders, VAEs)

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions