Skip to content

Give training its own GPU device - #10

Open
lanxinger wants to merge 4 commits into
mainfrom
port-pr-526-training-device
Open

Give training its own GPU device#10
lanxinger wants to merge 4 commits into
mainfrom
port-pr-526-training-device

Conversation

@lanxinger

@lanxinger lanxinger commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adapt ArthurBrussee/brush PR Give training its own GPU device ArthurBrussee/brush#526 through upstream head 65f19e0ada6221b6848bd9cda09b0a8ba83cf250.
  • Give training a lazy dedicated GPU device so viewer work no longer shares its queue.
  • Move displayed packed RGBA8 frames across devices through a host readback and viewer upload buffer.
  • Generalize custom kernels over CubeRuntime and update the pinned Burn, CubeCL, and Cubek revisions.
  • Make validation readbacks runtime-switchable so timing harnesses can disable synchronization.
  • Reach CubeCL through Burn and centralize the remaining direct burn-cubecl surface in brush-cube.

Fork-specific adaptations

  • Preserve native-MSL renderer paths, PPISP and appearance training, seeded training, LOD and refine behavior, JSONL metrics, iOS Metal selection, and robust PLY dataset detection.
  • Route fork-only appearance, native-MSL optimizer/min-scale, and loss-test CubeCL consumers through Burn or brush-cube.
  • Keep the Autodiff image-loss backward arm callable on the inner backend.
  • Make repeated initExisting calls idempotent.
  • Avoid a repository-wide typo exception by rewording the flate2 pin comment.

Upstream source: ArthurBrussee#526
Upstream head reviewed: 65f19e0ada6221b6848bd9cda09b0a8ba83cf250

Validation

  • cargo fmt --all -- --check
  • cargo check --workspace --all-targets --all-features
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo check --locked --workspace --no-default-features --all-targets
  • Portable WASM and native-MSL WASM-fallback checks on wasm32-unknown-unknown
  • Full Metal-backed all-feature workspace suite: 284 passed, 0 failed, 2 intentionally ignored
  • All-feature private documentation with warnings denied
  • Diff and whitespace checks
  • Egg dataset smoke: 24 masked views at max resolution 512, 10 training steps, successful final export in 4.48 s; output was a valid 70 MiB binary PLY with 313,214 splats

The ignored tests are the documented large host-Metal appearance dispatch regression (~1.5 GiB peak) and native-Metal launch-geometry soak. The local cargo-deny binary is unavailable; the upstream head cargo-deny check is green.

Adapt upstream PR ArthurBrussee#526 to the local native-MSL, appearance, LOD, metrics, iOS, and dataset paths. Keep validation readbacks switchable and update Burn/CubeCL/Cubek to the PR's pinned revisions.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 30, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-30T13:03:10.818538Z c136345 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c13634540e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/brush-process/src/lib.rs Outdated
connect_device(burn.clone());
// A JS host can call `init()` and `initExisting()`, or a dev-mode double
// mount can re-run setup. Whoever gets there first wins.
let _ = DEVICE.set(burn.clone().into());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Await registration of the host-provided device

tokio::sync::OnceCell::set is asynchronous, so assigning the returned future to _ drops it without ever storing burn. Consequently every initExisting(...) call appears successful, but the first later device() call opens the internal default device instead; training buffers then belong to that device and cannot be bound by the host WebGPU device as the brush-js API promises. Make this initialization path await set (or otherwise synchronously register the value) before returning success.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a false positive for the dependency version pinned by this PR. Cargo.lock resolves Tokio 1.53.1, where tokio::sync::OnceCell::set is synchronous: pub fn set(&self, value: T) -> Result<(), SetError>. Its implementation calls set_value before returning, so the host-provided device is stored before initExisting returns; no future is created or dropped here. The ignored Result only covers an earlier or in-flight initializer winning, consistent with the documented requirement that initExisting be called before anything touches the device. No code change is needed.

Adapt upstream 65f19e0 across fork-only appearance, native-MSL, loss, and benchmark paths. Keep burn-cubecl direct only in brush-cube.
Reject conflicting host-device initialization while preserving the Rust API. Avoid redundant viewer uploads, keep validation active in tests, adopt direct Burn CubeCL ownership, and add browser interoperability coverage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant