Skip to content

Add AR Wiener Storm example - #427

Merged
willeastcott merged 3 commits into
mainfrom
ar-wiener-storm
Aug 30, 2026
Merged

Add AR Wiener Storm example#427
willeastcott merged 3 commits into
mainfrom
ar-wiener-storm

Conversation

@willeastcott

Copy link
Copy Markdown
Contributor

A webcam AR example that pelts your MediaPipe-tracked head with floppy, physically simulated raw wieners. Whimsical on the surface, but it contributes several reusable pieces to the webcam AR script family.

What's in it

The example (ar-wiener-storm.html): wieners lob in from the hemisphere between your head and the screen on drag-aware ballistic arcs, bounce off an invisible head proxy, cast real shadows onto your face, vanish behind your head, and can be dodged — or batted away — because the physics world is anchored to the room, not your head. The HUD is a dignity meter (drains per hit, seeps back while you dodge) plus a deadpan announcer, driven entirely by app events. ?sim runs without a webcam, ?debug renders the head proxy for fit checks.

New reusable scripts:

  • tracked-head.mjs — face tracking with the frame flipped: the camera stays pinned at the origin of MediaPipe's camera space and the tracked head pose drives an entity instead. World-simulated content becomes dodgeable, where the base class's face-locked frame would drag it along with the head. Includes the nose-bridge anchor pin in the flipped frame. Builds on faceTracking, whose smoothed pose and sim orbit are promoted from private to protected.
  • head-shadow-catcher.mjs — grounds virtual objects on the real head: an invisible proxy that renders only the shadows cast onto it. The stock engine shadow catcher darkens the canvas multiplicatively, which cannot affect a DOM video behind a transparent canvas, so this one routes the engine's shadowCatcher accumulator into the canvas alpha as premultiplied black (GLSL + WGSL chunk overrides), gated by N·L so shadows never wrap past the terminator. It doubles as a depth occluder. Accepts any model as the catcher surface, with a cranium ellipsoid fallback.
  • video-ibl.mjs — low-frequency image-based lighting estimated from the live camera feed (in the spirit of mobile AR lighting estimation), so virtual objects pick up the room's real color temperature and brightness. Rebuilds scene.envAtlas into reused targets on a slow cadence; the declared sky remains the fallback when no camera runs.

New assets: wiener.glb (procedurally authored in Blender: bent capsule, six-bone chain, slick raw-pork material), canonical-head.glb (MediaPipe's canonical face mesh extended into a full cranium, so it registers with the tracked face and shadows bend around the nose and brow — Apache 2.0, sidecar included), and wiener-slap.mp3 (a wet slap, velocity-scaled at playback).

Reviewer notes

  • Each wiener is a chain of six capsule rigid bodies linked by 6dof spring joints, skinned one-to-one to the mesh's bones — the bending is physics, not animation. Physics steps at 180 Hz with raised solver iterations (both restored on destroy): head impacts spin the light segments past their joint limits within a single 60 Hz step otherwise.
  • Directional lights emit along the entity's negative Y axis, not forward — the light angles in the markup were read back from lookAt + rotateLocal(90, 0, 0).
  • The head proxy stays a pc-entity (model nested as a child) because entity: references resolve bare names against pc-entity elements only; a library-side follow-up for that is tracked separately.
  • face-tracking.mjs changes are doc-visibility only (three members promoted to protected); no behavior change, and all existing examples using it are unaffected.

Verified in ?sim on both WebGL2 and WebGPU (the catcher's chunk overrides ship in both shader languages); ESLint and the full test suite (1023 tests) pass.

🤖 Generated with Claude Code

willeastcott and others added 3 commits August 30, 2026 15:33
A webcam AR example that pelts the user's MediaPipe-tracked head with
floppy, physically simulated raw wieners. Beyond the comedy, it packs
several reusable pieces for the webcam AR script family:

- tracked-head.mjs: face tracking with the world anchored to the room
  instead of the head - the camera stays pinned at the origin of
  MediaPipe's camera space and the tracked head pose drives an entity,
  so world-simulated physics can be dodged by moving your head. Builds
  on faceTracking, whose smoothed pose and sim orbit are promoted to
  protected for subclasses.
- head-shadow-catcher.mjs: an invisible head proxy that renders only
  the shadows cast onto it, written into the transparent canvas alpha
  as premultiplied black so the darkening lands on the camera feed
  behind it. Gated by N.L so shadows never wrap past the terminator,
  and it doubles as a depth occluder. Uses canonical-head.glb, the
  MediaPipe canonical face mesh extended into a full cranium, so
  shadows bend around the nose, brow and cheeks.
- video-ibl.mjs: low-frequency image-based lighting estimated from the
  live camera feed, so virtual objects pick up the room's real color
  and brightness. Falls back to the declared sky when no camera runs.
- wiener-storm.mjs: the cannon. Each wiener is a chain of six capsule
  rigid bodies linked by 6dof spring joints, skinned one-to-one to a
  six-bone mesh, thrown on drag-aware ballistic arcs aimed at the head
  at launch. The storm escalates over time and reports hits, dodges
  and categories as app events for the page HUD (a dignity meter and
  a deadpan announcer).

Physics runs at 180Hz with raised solver iterations (restored on
destroy) because head impacts spin the light segments past their joint
limits inside a single 60Hz step. Lights are aimed via lookAt plus
rotateLocal(90,0,0) since directional lights emit along the entity's
negative Y axis.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- video-ibl: enable source mipmaps so the cubemap projection samples each
  texel once instead of 1024 times, and shrink the atlas to 128px with
  modest sample counts - the engine's 512px/1024-sample defaults burned
  tens of milliseconds of GPU per rebuild resolving detail a 64px
  low-frequency source does not have.
- video-ibl: the engine's equirect mapping puts the camera's view
  direction (-Z) at the panorama seam, not at u=0.5 (+Z), so the
  observed band was painted 180 degrees behind the camera. Center it on
  the seam and draw it wrapping across both edges.
- head-shadow-catcher: only emit the light0_direction facing gate when a
  shadow-casting directional light actually joined the catcher layer;
  with no qualifying light the uniform does not exist and the shader
  failed to compile.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@willeastcott
willeastcott merged commit 6cf31f0 into main Aug 30, 2026
3 checks passed
@willeastcott
willeastcott deleted the ar-wiener-storm branch August 30, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant