ASTRO BOT (PPSA21564) on Windows / AMD RX 6600 XT: GPU hang in the first 3D frames after the start-up logos (TDR, vkQueueSubmit -4) #6284

Activity

1. MarcZX10 commented on Oct 15, 2026

   

   MarcZX10

   on Oct 15, 2026

   Contributor

   More actions

   The ring dump narrows the stall to the last queue-0 frame but not to one draw or dispatch within that submission. In the current source, APS5_BATCH_CAP is the maximum recorded-work count before submitting (default 64), and APS5_TRACE_DRAW_PROGRAMS=1 logs draw program addresses. A warm-cache run with APS5_BATCH_CAP=1 APS5_TRACE_DRAW_PROGRAMS=1, keeping the same title/settings, should show the last individual work item that completes before the GPU hang. If the cap changes whether it hangs, that comparison would also help distinguish a single-command hang from an interaction/order issue.
2. Cleverdee commented on Oct 15, 2026

   

   Cleverdee

   on Oct 15, 2026

   ContributorAuthor

   More actions

   Ran it: same build (main db81992c5 + #4855), warm cache, APS5_BATCH_CAP=1 APS5_TRACE_DRAW_PROGRAMS=1, plus the same APS5_PM4_RING=64 APS5_STALL_WATCHDOG_SECS=3, TdrDelay=10.

   The cap does not change it. It hangs at the same point, only slower (13 min 53 s instead of ~4 min): recorded batch still running on the GPU after 5 s, [device-fault] result 0: No fault detected, vkQueueSubmit recorder: Vulkan result -4, event 4101.

   Where it stops. All traced draws before the hang belong to queue 0 serial 9679 (queue entry 5730, 25355 dwords): the main pass of the frame (first markers JsParticleUpdate, Main_0_0_0, ClearDistortionBuffer_0, blur_dir_shadow_0, ShadowVolume_0, CaptureCubemap_0, GiProbe_0). The same pass in the two previous frames (serials 9663 and 9671) has 218 draws with identical program pairs in identical order. In 9679 the trace stops after 67 draws, and those 67 match the earlier frames exactly:

     64  vertex/main=0x5008bed00 fragment/fragment=0x5008c0200
     65  vertex/main=0x148e7cd00 fragment/fragment=0x7ffbf9c98200   (rect-list draw, host fragment program)
     66  vertex/main=0x500644f00 fragment/fragment=0x500645900       <- last draw traced
     67  mesh/main=0xbe4bf100   fragment/fragment=0x500707b00       <- never decoded in 9679
     68  vertex/main=0x5008f0300 fragment/fragment=0x5008f1a00
   

   [draw-programs] is printed when the draw is decoded (Draw.cpp, before recording), so draw 67 (the only mesh-stage draw in that part of the pass) was never decoded and never reached the GPU. The game had already submitted serials 9682-9686 (Draw Geometry_1 .. Draw Shadow_1), so the queue worker was blocked inside 9679 between finishing draw 66 and decoding draw 67, waiting on work already on the GPU. With the cap at 1, the stuck batch should be draw 66 (0x500644f00 / 0x500645900), the rect-list draw 65, or a dispatch recorded between 66 and 67 (dispatches aren't in this trace). No program in the last 300 traced draws is new.

   Rect-list draws with a host fragment program appear at indices 9, 11, 65, 101 and 160 of each full frame; this run logs the usual Rect-list draw of shader 0x148e7cd00 has no rectangle for fragment shader 0x7ff...; preparing it at draw.

   Log attached (zipped, 12 MB unpacked): astro-cap1-trace-game.err.zip. Happy to run a dispatch trace or anything else around draw 66/67.

   AI-assisted: yes
3. MarcZX10 commented on Oct 15, 2026

   

   MarcZX10

   on Oct 15, 2026

   Contributor

   More actions

   One focused next capture: add APS5_TRACE_DISPATCH_IO=1 to the same warm-cache APS5_BATCH_CAP=1 APS5_TRACE_DRAW_PROGRAMS=1 run. In current Dispatch.cpp, this prints each compute shader address ([dispatch-io] shader ...); the draw-program trace does not include dispatches, so this can reveal whether a compute dispatch occurs between draw 66 and the point where decoding draw 67 blocks. APS5_TRACE_DISPATCH_IO=2 also prints user-data words and will be much noisier, so I would start with =1.
4. MarcZX10 commented on Oct 15, 2026

   

   MarcZX10

   on Oct 15, 2026

   Contributor

   More actions

   Thanks, the cap=1 trace confirms the worker stalls before decoding draw 67, but it still cannot tell whether a queued compute dispatch is involved. Could you run the same warm-cache setup once with APS5_TRACE_DISPATCH_IO=1 added, keeping APS5_BATCH_CAP=1, APS5_TRACE_DRAW_PROGRAMS=1, and the existing watchdog/ring settings? Please include the final [dispatch-io] shader lines around the last completed serial and the corresponding [gpu] execute/done serial lines if present. This should help identify the dispatch boundary without the much noisier user-data mode.
5. Cleverdee commented on Oct 15, 2026

   

   Cleverdee

   on Oct 15, 2026

   ContributorAuthor

   More actions

   Ran it: same build (main db81992c5 + #4855), same warm cache, APS5_BATCH_CAP=1 APS5_TRACE_DRAW_PROGRAMS=1 APS5_PM4_RING=64 APS5_STALL_WATCHDOG_SECS=3 APS5_TRACE_DISPATCH_IO=1, plus APS5_TRACE_GPU=1 for the execute/done serial lines. AMD driver is now 26.9.2 (32.0.21045.11001); the hang is the same on it. It took 66 min to get there with these traces (0.38 fps).

   Same stop: queue 0 serial 9543 (the main pass), last draw decoded vertex/main=0x500644f00 fragment/fragment=0x500645900 right after the rect-list draw, then recorded batch still running on the GPU after 5 s / 10 s, no fault, Vulkan result -4, event 4101. The watchdog did not print.

   No compute dispatch between draw 66 and the stall. The last [dispatch-io] shader line comes about 60 draws earlier:

   1036788 [gpu] 3963688.5 queue 0x0 waits 0x502704a00 == 0x1 (now 0x0)
   1036789 [gpu] 3963694.2 queue 0x0 waits 0x502704be0 == 0x1 (now 0x0)
   1036790 [dispatch-io] shader 0x500758900
   1036791 [dispatch-io] 1x1x1: texture 0x556760000+0x10000(1x1 f77 t7) nz=0.00 texture 0x5704f0000+0x10000(1x1 f77 t7) nz=1.00 storage 0x556760000+0x10000(1x1 f77 t7) nz=0.00 buffer(ro) 0x502dfbe60+0x10 ...
   1036792 [gpu] 3963727.9 queue 0x0 waits 0x502dfbe30 == 0xdeadbeef (now 0x0)
   

   What sits between the last draws is queue 0 WAIT_REG_MEMs, each logged and then satisfied (the next draw is decoded):

   1036852 [draw-programs] submission=9543 queue=0x0 count=2 vertex/main=0x5008bed00 fragment/fragment=0x5008c0200
   1036853 [gpu] 3964476.5 queue 0x0 waits 0x502deed60 == 0xdeadbeef (now 0x0)
   1036854 [gpu] 3964482.6 queue 0x0 waits 0x5027096b0 == 0x1 (now 0x0)
   1036855 [gpu] 3964488.1 queue 0x0 waits 0x5027097a0 == 0x1 (now 0x0)
   1036856 [gpu] 3964494.4 queue 0x0 waits 0x502709da0 == 0x1 (now 0x0)
   1036857 [draw-programs] submission=9543 queue=0x0 count=2 vertex/main=0x148e7cd00 fragment/fragment=0x7ff8e2bf8200   (rect-list)
   1036858 [gpu] 3964512.3 queue 0x0 waits 0x50270a0e8 == 0x1 (now 0x0)
   1036859 [draw-programs] submission=9543 queue=0x0 count=2 vertex/main=0x500644f00 fragment/fragment=0x500645900
   1036860 [gpu] recorded batch still running on the GPU after 5 s
   1036861 [gpu] recorded batch still running on the GPU after 10 s
   1036862 [device-fault] result 0: No fault detected
   

   So the wait between the rect-list draw and draw 66 completed, draw 66 was decoded, and the next GPU completion never came. With the cap at 1 that points at draw 66's batch (or the rect-list draw just before it, if that wait didn't depend on it).

   Last execute / done serial per queue before the stall:

   queue	last done	still executing
   0x0	9542 (t=3961565.0)	9543 since t=3961571.4, 25k-dword main pass
   0x40	9537 (t=3963564.7)	9545 since t=3963580.5
   0x48	9539 (t=3961049.0)	9547 since t=3961058.4
   0x3a	6928 (t=2264775.0)	none

   The [dispatch-io] lines don't name their queue, so I can't tell from this log whether 9545 / 9547 got any dispatch onto the GPU after the lines above; the excerpt has every line from the last ~340 before the stall.

   Attached: astro-dispatch-tail.txt (last ~340 lines before the stall) and astro-dispatch-trace-game.err.zip (full log, 90 MB unpacked).
6. MarcZX10 commented on Oct 17, 2026

   

   MarcZX10

   on Oct 17, 2026

   Contributor

   More actions

   I opened PR #7010, a diagnostic-only change that adds the queue ID and submission serial to each APS5_TRACE_DISPATCH_IO shader line. It does not claim to fix the GPU hang. If it lands, the same capture can identify whether later dispatches came from queues 0x40 or 0x48 while queue 0 was stalled.