Skip to content

agc_driver_tests: rare heap-corruption exit (0xc0000374) on Windows remains after #2976 #3171

Closed

Description

@SleglJan

What

On Windows, agc_driver_tests occasionally ends with STATUS_HEAP_CORRUPTION (0xc0000374); @Yuva1024's loop reported it as exit 127. #2976 removed one candidate cause and closed #2891, but the exit still occurs with that change applied. I'm opening this as @adriadam10 asked on #2976.

Data so far (all from #2891)

The exit still happens with #2976 applied, so a cause remains. Whether the use-after-free was ever one of the causes is open. At about 0.3 %, the Windows rate is too low to tell whether #2976 lowered it.

Next step

A Windows run with full page heap, as queeeef suggested: gflags /p /enable agc_driver_tests.exe /full. It turns a bad heap write into an immediate access violation, which the crash reporter prints with a backtrace. I have no Windows machine, so I can't run it myself.

Related:

AI-assisted: yes (Claude Code)

Activity

  1. mrcrapto commented on Oct 11, 2026

    @mrcrapto
    Contributor

    Additional Windows reproduction data on main revision 961b25fe24582361833a90b912eff5793550afdf, which includes the registration-preparation shutdown drain:

    • Windows 11 Pro 10.0.26300; WinLibs GCC 15.2.0 r7; Release, BUILD_TESTING=ON.
    • Intel i7-13620H, 16 logical processors, 16 GB RAM.
    • NVIDIA RTX 4060 Laptop GPU, driver 32.0.15.9649.
    • agc_driver_tests.exe alone: 400 runs, four processes in parallel, no additional CPU stress. Vulkan required; shader disk cache disabled.
    • 400/400 exited 0, selected the RTX 4060 and printed the test completion marker. No runner timeouts. Campaign lasted about seven minutes on 11 October 2026. Per-run stdout/stderr and raw Windows exit codes were retained.

    I did not enable full page heap: the session was non-elevated and Windows Debugging Tools were unavailable. This is therefore a standard-heap reproduction result, not the requested page-heap diagnosis. Zero failures in this campaign do not establish that the rare corruption is fixed. No faulting-write trace or new fix is claimed.

    AI-assisted: yes (validation scripts and reporting); tests ran on local hardware.

    Validation evidence and reproduction scripts (published by @mrcrapto).

  2. Silverash-i8 commented on Oct 11, 2026

    @Silverash-i8
    Contributor

    A possibly useful trigger for this, from my Windows machine tonight: after device losses, the heap-corruption exit stops being rare.

    Setup: Windows 11 Home (26200), WinLibs GCC 15.2.0 r7, Release, main 961b25f (plus #3141, which only touches the relinker), Arrow Lake iGPU (ANYPS5_GPU=Intel, driver 101.8991).

    Earlier tonight, while testing #2985 on #2832, the iGPU went through a burst of Windows GPU watchdog resets (LiveKernelEvent 141). Since then agc_driver_gated_dispatch_capture mostly loses the device on it, even on plain main, where it passed before. In that state:

    • Through ctest, 23 runs on 961b25f and d31b57a: 2 passed, 14 failed with device lost, 6 exited 0xc0000374, 1 segfaulted.
    • Running agc_driver_gated_dispatch_capture_tests.exe directly 8 times just now: 7 exit 1 (device lost), 1 exits 127 (0xc0000374, as in Yuva1024's loop).

    The last lines differ between the two kinds of run:

    # exit 1
    [gpu] queue 0x28 dropped 1 queued labels: their record failed
    the gated submissions did not finish
    [gpu] unit shadow teardown: AGC graphics: vkQueueSubmit recorder: Vulkan result -4
    AGC graphics: vkWaitForFences recorder: Vulkan result -4
    
    # exit 127 (heap corruption)
    [gpu] queue 0x28 dropped 1 queued labels: their record failed
    (nothing after this)
    

    So the corrupting write seems to happen on the path after a recorder submission fails and the queue drops its labels, before the test reports and the teardown runs. That might also be what the rare cases hit, if a submission occasionally fails there. The RTX 5060 in the same machine passes gated_dispatch_capture 10/10 with no device loss.

    I can't run gflags /p /enable ... /full right now (it needs an elevated prompt), but I can once I'm back at the machine, and the device-lost state makes it reproducible here in a few runs.

    AI-assisted (Claude Code); I ran all of this on my machine.

  3. mrcrapto commented on Oct 11, 2026

    @mrcrapto
    Contributor

    Following the submission-failure lead, I isolated a deterministic cleanup defect in Recorder::Submit(): after moving out the open batch, a failing vkEndCommandBuffer or vkQueueSubmit destroys that batch while leaving its label-table entries, completion counts and Vulkan handles behind. The labels can therefore retain pointers to the destroyed batch.

    Patch: #3298 (commit). The PR was automatically closed by the current intake policy; the commit remains available for review or cherry-picking.

    The patch discards the failed batch without running its completions, preserves earlier submitted batches, and waits for device idle before releasing resources after queue-submit device loss, following the Vulkan lost-device rules. The initial two-case injected regression failed on unchanged code with 14 failed assertions. All six final failure variants pass, including both allowed idle-wait results and preservation of an earlier batch.

    On the final de129cfd base, NVIDIA RTX 4060 Laptop agc_driver and gated-dispatch capture pass. On the initial d7cfa728 base, Intel UHD gated-dispatch capture returned device loss twice with the patch, while one unchanged-code comparison passed. The Intel cause remains unresolved. Full default Windows compilation, the relevant runtime library/test targets and the conventions check passed.

    This is not a confirmed fix for the reported heap corruption or Intel device loss; I have no page-heap faulting-write trace. It is a separately reproduced cleanup defect on the error path you identified.

    Regression and validation logs. AI-assisted: yes; tests ran locally.

  4. Silverash-i8 commented on Oct 11, 2026

    @Silverash-i8
    Contributor

    @mrcrapto thanks, I ran your commit in the device-lost state I described, since it reproduces here in a few runs. Same machine and Arrow Lake iGPU, agc_driver_gated_dispatch_capture_tests.exe run directly 24 times each:

    Build exit 1 (device lost) exit 127 (0xc0000374) passed
    de129cfd (your base) 22 2 0
    e487ff00 (your commit) 18 5 1

    So in this state the heap-corruption exit is still there with the patch. 2 vs 5 out of 24 is noise at this sample size, so I'm not reading anything into that difference. The cleanup defect you found looks real to me; it just isn't the one this repro hits.

    Every 0xc0000374 run, with and without the patch, ends on the same line, and the test's own the gated submissions did not finish never prints:

    [gpu] queue 0x28 dropped 1 queued labels: their record failed
    

    That message comes from the Clear guard's destructor in Driver::recordDeferredLabels (Execution/src/Driver/Synchronization/DeferredLabels.cpp:104), so it runs while an exception from the label loop (WriteLabelOnGpu, WaitIdle, GuestMemory::Write or noteLabelStore after the device is lost) is unwinding. Right after the print it does labels.clear(), Recorder::CloseLabelGroup(0) and Recorder::ForgetQueuedLabels(). Since stderr is unbuffered and nothing else gets printed, the corrupting write is probably in one of those three steps, or in a destructor that runs during the same unwind. That's a different path from the Recorder::Submit() one your commit fixes, which may be why it doesn't change this.

    The page-heap run is still on my list for when I have an elevated prompt.

    AI-assisted (Claude Code); I ran all of this on my machine.

  5. Silverash-i8 commented on Oct 11, 2026

    @Silverash-i8
    Contributor

    Correction to my last comment: I added probe prints, and the corruption isn't in those three cleanup steps. On de129cfd, first with prints after labels.clear(), CloseLabelGroup and ForgetQueuedLabels in the Clear destructor (16 runs, 3 heap-corruption exits), then also at the queue worker's catch (...) entry, after the flip failures and after ReportFailure in QueueWorker.cpp (26 runs, 2 heap-corruption exits). All 5 heap-corruption runs printed every probe their build had, same as the device-lost runs that exit normally.

    So the destructor finishes, the exception reaches the worker's catch (...), and ReportFailure returns. The process dies after that and before the test's own the gated submissions did not finish, so somewhere between the worker reporting the failure and the main thread getting it back. The heap is only checked at the next allocation or free, so that's where it's noticed, not necessarily where the bad write happens.

    So the page-heap run is really the next step; prints won't narrow it much further. I'll do it when I have an elevated prompt, with this device-lost state as the reproducer.

    AI-assisted (Claude Code); I ran this on my machine.

  6. zaydzaari commented on Oct 11, 2026

    @zaydzaari
    Contributor

    Windows full page heap exposed a separate reproducible AGC TLS use-after-free during this investigation. Refs #3171; I have not established that it explains the original rare 0xc0000374 exit.

    Two debugger runs of the unchanged baseline (06da1176) failed in std::vector<Driver::PendingWait>::~vector. In the second run the object address was 0x2415b6c0fe8; !heap -p -a identified its allocation as already freed. The recorded free stack was:

    ucrtbase!free_base
    libgcc!emutls_destroy+0x33
    winpthreads!pthread_cleanup_dest
    pthread_create_wrapper
    

    The subsequent access came through vector<PendingWait>::~vector -> run_dtor_list -> tls_callback -> LdrShutdownThread. This establishes that emutls cleanup freed the vector object's storage before the DLL TLS callback invoked its C++ destructor. The evidence supports a use-after-free of the object itself; it does not establish a double free or heap overwrite.

    The authored correction is c32d7e3f92fd34f2e3da14510dbfcb9ae0dc0076, on fix/agc-thread-local-teardown-current, based on upstream 0f863c10. It applies the existing ShaderRecompiler::ThreadOwned and HostThreadLocal helpers to the remaining affected wait vector, dispatch map, image memo, recorder scratch vectors, and precise-wait timer. This follows the lifetime approach used in #590. The seven-file patch also adds a MinGW object-symbol regression guard against __cxa_thread_atexit registrations, using a generated manifest to avoid Windows command-line limits.

    Local Windows 11 / MinGW-w64 GCC 15.2.0 Release validation:

    • Baseline: 2/2 debugger runs reproduced the destructor failure; the new guard rejected five baseline object files.
    • Final patched revision: 18/18 selected CTest tests passed, including the guard; 5/5 full-page-heap debugger runs passed with no detected fault and debugger exit 0.
    • agc_driver and agc_driver_registration_prepare_shutdown: each passed 10/10 repetitions. Conventions and Git whitespace checks passed.

    The guard can be run with ctest --test-dir <build> -R '^agc_driver_thread_local_destructors$' --output-on-failure. These results eliminate the demonstrated TLS failure in the tested cases, but do not prove complete resolution of #3171 or the reported device-loss scenario. Full CI and independent Linux validation remain outstanding.

    Claude Code and Codex assisted with implementation, review, and testing. Would a maintainer consider reviewing or carrying this patch, preserving the original authored commit where possible?

  7. screamm commented on Oct 11, 2026

    @screamm
    Contributor

    More data points: on this Windows 11 Pro 26220 host (WinLibs GCC 15.2.0 r7, Release, RTX 4070 driver 32.0.15.9621, 16 CPUs) agc_driver hit the 0xc0000374 exit in about one of three ctest -j8 suite runs today across unrelated branches on main 0c617bae to a11c8f7f (six runs, no message before the exit, roughly 4 to 5 s in); run alone it failed once on the first repeat and then passed 3/3 twice on the same binaries. So the rate is higher than rare under parallel load here. Written with Claude Code assistance.

  8. AlxcBTW commented on Oct 11, 2026

    @AlxcBTW
    Contributor

    @zaydzaari's finding is the cause, and I think it also explains the rare suite exits. I got here from #2980 (agc_driver_draw_cache_reuse crashing on Windows + NVIDIA) and ended up at the same stack.

    Reproduction without page heap. Started under cdb, which turns on the debug heap's tail and free checks, agc_driver_draw_cache_reuse stops with c0000374 in every run: 127 of 127, on Windows 11, WinLibs GCC 15.2.0 and an RTX 5070 Laptop. It stops as Driver::stop joins the queue worker:

    pthread_create_wrapper → LdrShutdownThread → tls_callback → run_dtor_list → free

    That is after winpthread's key cleanup already ran emutls_destroy. A ten-line DLL with a thread_local std::vector used from a std::thread fails the same way, and passes once the vector lives in HostThreadLocal. So this is MinGW's TLS teardown order, and any worker thread that ends can hit it.

    Why it fits the reports here.

    • @screamm's one-in-three ctest -j8 runs: every agc_driver process ends its workers, and load decides whether the stale free lands on a block in use.
    • @Silverash-i8's device-lost repro: Driver::run returns from its catch (...) right after ReportFailure, so the worker ends exactly where the process died. I haven't reproduced that state, so that part is a fit, not a proof.

    The fix, extended. The same pattern is outside the driver too:

    • libkernel: directory path, UDP waker, UUID random_device;
    • libSceRandom and libScePosixForWebKit;
    • PreciseSleepUs's timer, whose destructor also closed a stale handle.

    AlxcBTW/AnyPS5 fix/thread-local-teardown moves the driver objects from @zaydzaari's c32d7e3 plus these to HostThreadLocal. It adds prx_thread_local_destructors, a MinGW-only test that checks all 122 built modules for __cxa_thread_atexit registrations outside libc's guest destructors. On main it fails on libSceRandom.

    Results with it:

    @zaydzaari found it first, so if you'd rather extend c32d7e3, take whatever helps from the branch.

    AI-assisted: yes (Claude Code)

  9. Amoenus commented on Oct 11, 2026

    @Amoenus
    Contributor

    Additional deferred-undecodable serial reproduction on Windows (2026-10-11)

    Reproduced STATUS_HEAP_CORRUPTION (0xc0000374) in a different small execution case, agc_driver_prepared_shaders_deferred_undecodable / agc_driver_prepared_shaders_tests.exe --deferred-undecodable.

    • Windows 11 Enterprise 26200; WinLibs GCC 15.2.0 r7; Release, BUILD_TESTING=ON; RTX 5080, NVIDIA driver 32.0.16.1062. Native Vulkan required and shader disk cache disabled.
    • Exact tested integration: Amoenus/AnyPS5 f7a2bf691, based on upstream a11c8f7. It includes our kernel/math/D64 and experimental sampled-MSAA changes. This is not an unchanged-main control and does not establish their involvement or independence. The PreparedShaders.cpp test source is unchanged from a11c8f7.
    • Broad relevant run, parallel=2 / repeat until-fail:5: this case passed twice, then failed on repetition 3 in 0.39s. The other 72 selected test groups passed all five repetitions.
    • Subsequent isolated repeat until-fail:20 / parallel=1, same binaries: four passes, then the same heap-corruption exit on repetition 5 in 0.36s. No runner timeout. Thus concurrent test processes are not required in this reproduction; these short campaigns do not measure a stable failure rate.

    Last line in the serial failed run:

    [Shaders/ShaderRegistry.cpp:565 operator()] Shader 0x3811bff400 could not be prepared at registration (unsupported MIMG opcode 0xe5); preparing it when it is used
    

    The test's final deferred undecodable shader tests passed marker is absent. There is no device-loss message in that failed log. This identifies the last observed output, not the faulting write or cleanup function.

    Reproduction after building this integration:

    $env:ANYPS5_GPU='RTX 5080'
    $env:ANYPS5_REQUIRE_VULKAN='1'
    $env:ANYPS5_NO_SHADER_CACHE='1'
    $env:ANYPS5_REQUIRE_MSAA_ALL='1'
    ctest --test-dir build-review -R '^agc_driver_prepared_shaders_deferred_undecodable$' --repeat until-fail:20 --parallel 1 --timeout 30 --output-on-failure

    At that checkpoint there was no debugger/full-page-heap trace or corrected-build comparison. I have read the independently traced TLS findings and fork fixes above; I have not validated them against this case, and am not claiming a shared cause. Reporting here to avoid a duplicate heap-corruption issue. Raw CTest logs/XML are retained locally; the exact failure excerpts and commands are supplied above. No PR created. AI-assisted investigation/reporting: Codex.

    Exact-main control follow-up

    On unchanged main a11c8f7f, separate serial --repeat until-fail:20 --parallel 1 --timeout 30 campaigns now reproduce 0xc0000374 in agc_driver_prepared_shaders_debug, agc_driver_prepared_shaders_deferred_undecodable and agc_driver_prepared_shaders_deferred. Same supported WinLibs r7/RTX 5080 environment, Vulkan required and cache disabled; no MSAA integration or game execution required. These are exit-signature matches, without a faulting-write trace or shared-cause claim. Independent #3661 peer/main controls separate the passing focused tests from the remaining heap failures. Raw commands, logs and XML are retained locally.

    Corrected integration follow-up (2026-10-12)

    The three production changes from #4029 are now applied locally on pinned maina672 integration, with author credit. Its original module-destructor scanner passes all122 unpatched modules; the preserved original AGC module fails it. All five selected heap cases pass ×20 (100/100), and the full suite records863pass/0fail/5skip. Commands, exact commit scope and limitations. Before applying it, this machine again reproduced heap exits in parallel debug and sequential deferred-undecodable runs. No local debugger teardown trace, Linux runtime or unchanged wholePRhead claim; finite clean campaigns do not prove all rare crashes absent. No push/new PR.

  10. Silverash-i8 commented on Oct 11, 2026

    @Silverash-i8
    Contributor

    Here's the page heap run I promised. It points at @zaydzaari's finding for the 0xc0000374 exits in my repro too (@AlxcBTW could only call that a fit), and AlxcBTW's branch makes the fault go away.

    Setup. Windows 11, WinLibs GCC 15.2.0 r7, Release, main 0f863c1 (the base of AlxcBTW's branch), agc_driver_gated_dispatch_capture_tests.exe on the Arrow Lake iGPU (ANYPS5_GPU=Intel, ANYPS5_NO_SHADER_CACHE=1). That's the run that gave 0xc0000374 in roughly one of five to eight runs after the device-lost path. No gflags or cdb here, so I set full page heap through Image File Execution Options for that executable (GlobalFlag 0x02000000, PageHeapFlags 3) and read the test's own crash report.

    On main: 4 of 4 runs fault at the same instruction. Under page heap none of them reached the device loss: each logs a WAIT_REG_MEM ... timed out after 1000ms and then faults while the driver shuts down, before the test reports a result:

    FATAL: unhandled exception 0xc0000005
      rip libSceAgcDriver.prx+0x6399d3
      read of 0x000001fe2e5eafe8
      frame pointer chain:
        #0 libSceAgcDriver.prx+0x5cc906
        #1 ntdll.dll+0xe83fb
        #2 ucrtbase.dll+0x2cd30
    

    With the module's symbols (nm, nearest preceding symbol):

    • +0x6399d3 is std::vector<AgcDriver::DriverDetail::Driver::PendingWait>::~vector() +0x3, the static thread_local std::vector<PendingWait> waits in WaitMemory.cpp:57
    • +0x5cc906 is tls_callback +0x16d, and the stack also holds run_dtor_list +0x43 (both tls_atexit.c)

    The read hits memory page heap has already freed. In all four reports the main thread is in Driver::Shutdown() and another thread is in ReleaseThreadMain(), so the faulting thread is a worker ending at shutdown. That's the same ~vector -> run_dtor_list -> tls_callback path zaydzaari traced. Without page heap the stale free usually goes unnoticed, which fits the rare 0xc0000374 I saw at the end of runs, after the device loss, and screamm's suite-level rate.

    With AlxcBTW's fix/thread-local-teardown (20dfb17), page heap still on: 6 of 6 runs end without a fault. Three passed. Three hit the device loss (dropped 1 queued labels, "the gated submissions did not finish", exit 1) and still shut down cleanly.

    Then without page heap, 20 runs of the branch: 18 passed, 2 took the device-lost path and exited 1, none exited with 0xc0000374. On its own that's weak, since the iGPU passes most runs today, so few of them reach the exit that corrupted the heap. The page heap comparison is the real evidence.

    I haven't looked at @Amoenus's --deferred-undecodable case; it isn't covered by any of the above.

    AI-assisted (Claude Code); I ran all of this on my machine.

  11. kr4y4 commented on Oct 11, 2026

    @kr4y4
    Contributor

    Another Windows reproduction, and a check of @AlxcBTW's fix/thread-local-teardown (20dfb17).

    Setup: Windows 11 (26200), WinLibs GCC 15.2.0 r7, Release, an RTX 4070 Laptop GPU and an Intel Raptor Lake-S iGPU. Each executable is started under the WinLibs gdb (gdb -batch -ex run), which turns on the debug heap's checks.

    On main a11c8f7 (RTX 4070), these stop with Critical error detected c0000374 every time:

    • agc_driver_dword_pattern_fill
    • agc_driver_draw_cache_reuse
    • agc_driver_prepared_shaders --deferred
    • agc_driver_compute_user_data_calls

    Every stack is _endthreadex → LdrShutdownThread → tls_callback → run_dtor_list → free in libSceAgcDriver.prx, the same as above. A test that only calls Recompile and VulkanDevice::Dispatch, with no driver worker threads, is clean. libSceAgcDriver.prx has 121 __emutls_v.* objects; the ones that own memory include Recorder::refreshDrawSnapshot's five vectors and Driver::pendingWaits's waits.

    Without a debugger, the bug shows up as rare exits. With 8 copies of agc_driver_prepared_shaders --deferred started at once on the RTX 4070, 2 of 64 runs exited with 0xc0000374 (exit 127 in Git Bash) on main. The same happened in the --registered mode of agc_driver_compute_user_data_calls (#3294), 2 of 64 runs.

    On the branch:

    • Under gdb, dword_pattern_fill, draw_cache_reuse and prepared_shaders --deferred report no heap corruption on either GPU.
    • compute_user_data_calls and its --registered mode are clean on the RTX 4070.
    • Every run exits 0.
    • The same 8-way run of --deferred: 0 failures in 64 runs.

    Written with AI assistance.

  12. AlxcBTW commented on Oct 11, 2026

    @AlxcBTW
    Contributor

    @kr4y4 thanks, that's a clean check: four tests that fail every time under gdb on main are clean on the branch, and 0 of 64 against 2 of 64 for the 8-way run.

    I'll open fix/thread-local-teardown as a PR as soon as #3546 is merged; until then the intake rules allow me one open PR. If someone wants to carry it sooner, 20dfb17 is ready.

    AI-assisted: yes (Claude Code)

  13. AlxcBTW commented on Oct 11, 2026

    @AlxcBTW
    Contributor

    Opened as #4029 (fix/thread-local-teardown-2, rebased onto main a11c8f7), with @Silverash-i8's and @kr4y4's checks linked. Thanks both.

    AI-assisted: yes (Claude Code)

  14. oneandonlydean commented on Oct 12, 2026

    @oneandonlydean
    Collaborator

    Closing: fixed by #4029 (thread_local objects with destructors no longer run in module teardown on Windows) together with #3661. Reopen with the stack if 0xc0000374 still shows up on a build that has both.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions