feat: wire video-control attribute through to Unity VideoPlayer

[claude]

Requested by mwfarb · Slack thread

Before: publishing a video-control attribute to a scene had no effect at all in the Unity client. The dispatch arm was commented out (Runtime/ArenaClientScene.Messaging.cs:316), ArenaUnity.ApplyVideoControl did not exist, and ArenaVideoControl.ApplyRender() was the empty // TODO: Implement this component if needed stub — so none of the 8 schema fields reached Unity, and the component was never even added to the object.

After: video-control is dispatched like its neighbours. The component is added to the object, its json is populated from the wire and shows in the Inspector, and looping, autoplay/pause and volume are applied to the VideoPlayer on the object that holds the video. A dependency that has not arrived yet no longer drops the update: RegisterAssetCallback re-arms apply when the video asset lands, and a named video_object or a VideoPlayer that does not exist yet is retried from Update() for a bounded window instead of being lost for the rest of the session.

It makes video-control a live inbound attribute in the Unity client for the fields that have a clean VideoPlayer counterpart.

How: ArenaMaterial.AttachMaterialTexture (Runtime/Components/ArenaMaterial.cs:612-636) already owns the VideoPlayer — it creates one on the GameObject carrying the Renderer, points it at the local asset, sets isLooping/playOnAwake and calls Play(). Rather than stand up a competing player, ApplyRender() resolves the target object (video_object via ArenaClientScene.Instance.arenaObjs, falling back to this object), hands video_path to AttachMaterialTexture through the same checkLocalAsset/RegisterAssetCallback pair ArenaMaterial uses for src, then reads back vobj.GetComponent<UnityEngine.Video.VideoPlayer>() and sets the fields on it. Because the attach happens first, our isLooping/playOnAwake/volume/pause writes land after the unconditional Play() inside AttachMaterialTexture. A named object that is missing, or a VideoPlayer that does not exist yet, no longer returns for good: ApplyRender() records the missing dependency and Update() sets apply = true again on following frames until it appears, giving up after a 30 s wall-clock window with a single warning. Where the player is still waiting on material.src, RegisterAssetCallback is armed once as well, so a download slower than that window still re-applies. Both paths terminate: the retry is bounded by wall clock, and the asset callback is single-shot (Runtime/ArenaClientScene.Assets.cs:158-161 removes it as it fires).

Closes #224

Field mapping

field	status	notes
video_object	mapped	resolved through ArenaClientScene.Instance.arenaObjs (same idiom as ArenaPhysxJoint.Target); falls back to this object when unset
video_path	mapped	passed to ArenaMaterial.AttachMaterialTexture, which owns the VideoPlayer; RegisterAssetCallback re-arms apply if the asset has not arrived
video_loop	mapped	VideoPlayer.isLooping
autoplay	mapped	VideoPlayer.playOnAwake, plus Play()/Pause() so a later autoplay: false actually pauses
volume	mapped	audioOutputMode = Direct then SetDirectAudioVolume(0, volume), guarded on controlledAudioTrackCount > 0. Direct is the only output mode where the wire's scalar volume has a counterpart without inventing an AudioSource; worth a look during review
frame_object	left as // TODO:	not an object — the schema field is the URL of a thumbnail image (Runtime/Schemas/Attribute/ArenaVideoControlJson.cs:32-34, tooltip "URL of a thumbnail image"). Showing it needs its own image load and a swap to the video on play — not wiring
anyone_clicks	left as // TODO:	needs click-to-toggle-playback routing through ArenaClickListener, whose own ApplyRender() currently throws NotImplementedException and is not reached (see #220). Deliberately out of scope here
cleanup	marked // N/A:	"remove HTML5 video and img assets from DOM on object removal" — there is no DOM; Unity destroys the VideoPlayer with the GameObject

How to test in the Editor

1. Put a video texture on an object — publish an object with material: { src: "store/users/<you>/videos/<clip>.mp4" } so AttachMaterialTexture creates the VideoPlayer, or publish video-control with video_path set and let this change create it.
2. Confirm the object now has both an ArenaVideoControl component with the wire values in its Inspector and a UnityEngine.Video.VideoPlayer.
3. Publish material and video-control together in one message, once as {"material": …, "video-control": …} and once with the keys the other way round. The settings should land either way — that is the ordering race, and it is now retried on a following frame rather than dropped.
4. Publish video-control with autoplay: true — playback should start. Publish again with autoplay: false — it should pause where it is (this is the pause path; the schema has no separate pause field). Note the open question above: on a material.src video, a video-control publish that omits autoplay pauses it.
5. Publish volume changes (0, 0.5, 1) and confirm the audio level follows. Note the VideoPlayer's audio track count is only known after prepare, so a volume sent before the clip is prepared may need one more publish to take.
6. Seek cannot be exercised: ArenaVideoControlJson has no time/seek field (the 8 fields are frame_object, video_object, video_path, anyone_clicks, video_loop, autoplay, volume, cleanup), so there is nothing on the wire to map to VideoPlayer.time. Worth confirming whether the ARENA schema is expected to grow one.
7. Publish video_loop: false and confirm playback stops at the end instead of restarting.
8. Publish video-control with video_object naming an object that does not exist yet, then publish that object within 30 s — the settings should land as it arrives, where before the message was dropped for the session. Then name a video_object that never arrives and confirm exactly one video-control gave up waiting for … warning after about 30 s, not one per frame.

Untested

Stated plainly: this is untested. There is no Unity Editor and no Unity licence in the environment this was written in, and no CI will run on this PR — the workflow that adds pull_request triggers exists only on the unmerged draft #183. The diff was read adversarially (assembly-internal access to AttachMaterialTexture, fully-qualified UnityEngine.Video.* so no new using is needed, ArenaClientScene resolved from the enclosing namespace to avoid the ArenaUnity namespace/class CS0426 collision), but it has not been compiled. Please build before believing any of it.

Notes for review

• Base-Update() hiding (fix: four components hide ArenaComponent.Update(), so repeat material-extras / modelUpdate / gltf-morph messages are dropped #220): checked, and ArenaVideoControl does not follow that pattern — it declares no private new void Update(). It now does override Update(), but as protected override that calls base.Update() (Runtime/Components/ArenaComponent.cs:25-32) before its own re-arm check, so ApplyRender() is still reached on every inbound update. This component is unaffected by fix: four components hide ArenaComponent.Update(), so repeat material-extras / modelUpdate / gltf-morph messages are dropped #220. (The anyone_clicks work would run into it, via ArenaClickListener.)
• Merge order matters. Two sibling drafts touch the same files: one adds the two physx joint appliers to Runtime/ArenaUnity.cs and arms to the same dispatch table, and another changes the visible guard in UpdateObjectMessage. The hunks here are kept tight — one line uncommented in place in the dispatch table, one applier appended after ApplyModelUpdate just before the // wire objects divider — but whichever lands second will likely need a trivial rebase in those two spots.
• Material ordering, corrected. An earlier version of this description blamed the order of the case arms in the dispatch switch. That was wrong: the arms are a lookup, so their textual order means nothing. The order comes from the wire message. UpdateObjectMessage parses the inbound data into jData (Runtime/ArenaClientScene.Messaging.cs:66) and walks it with foreach (var result in jData) (:259), so attributes are handled in whatever JSON key order the publisher sent — video-control (:316) can be reached before or after material (:327). The appliers do no work of their own either: ArenaUnity.ApplyVideoControl adds the component, deserializes the json and sets apply = true (Runtime/ArenaUnity.cs:418-424), and the actual rendering happens later, in each component's Update(), in Unity's component order. So the real hazard is that ArenaVideoControl can render before ArenaMaterial has created the VideoPlayer — which the bounded retry above now picks up on a following frame instead of dropping. Setting video_path still sidesteps the race by creating the player here.
• Open question for the maintainer — autoplay defaults to false and anyone_clicks is unmapped. defAutoplay = false (Runtime/Schemas/Attribute/ArenaVideoControlJson.cs:78), while AttachMaterialTexture starts a material.src video with playOnAwake = true and an unconditional Play() (Runtime/Components/ArenaMaterial.cs:612-636). Attaching video-control to an object whose video came from material.src, without autoplay: true on the wire, therefore pauses a clip that was playing — and since anyone_clicks is not wired, there is no click-to-play to start it again. No behaviour was changed for this; it is a product decision, not something to guess at. The options are to leave it wire-faithful, to touch playback only when autoplay is explicitly present in the message, or to wire anyone_clicks first. Please pick one.

---

Generated by Claude Code