Skip to content

fix(agc): start the shader preparation threads at the first job - #5196

Closed
nikowolf123 wants to merge 2 commits into
boykopovar:mainfrom
nikowolf123:fix/agc-prepare-pool-shutdown-threads
Closed

nikowolf123 wants to merge 2 commits into
boykopovar:mainfrom
nikowolf123:fix/agc-prepare-pool-shutdown-threads

Conversation

@nikowolf123

Copy link
Copy Markdown
Contributor

What

On Windows, agc_driver_resolution_shutdown and agc_driver_wait_shutdown sometimes time out because the test process never finishes exiting. It spins on one core, and taskkill /F cannot end it. The Windows runners show this in run 38198064871 (agc_driver_resolution_shutdown) and run 38186720056 (agc_driver_wait_shutdown). Both report Timeout 10.01 sec after the test had printed everything it prints when it passes: agc_driver_wait_shutdown its last line, unsatisfied label waits cancelled cleanly at shutdown, and agc_driver_resolution_shutdown only the libc startup line.

I attached gdb to stuck processes. Apart from the thread gdb adds, one thread is left, with this stack (the libwinpthread-1.dll and libstdc++-6.dll frames are named from the DLLs' export tables):

pthread_spin_lock                                   libwinpthread-1.dll
pthread_cond_destroy                                libwinpthread-1.dll
std::condition_variable::~condition_variable()      libstdc++-6.dll
AgcDriver::DriverDetail::Driver::~Driver()          libSceAgcDriver.prx
AgcDriver::DriverDetail::Driver::Get()::driver      (static destructor, run at DLL_PROCESS_DETACH)

The Driver member being destroyed was never waited on, so it still holds winpthreads' static initializer. For such a condition variable pthread_cond_destroy takes winpthreads' global cond_locked spinlock, the same lock that guards the first wait on every statically initialized condition variable. In every stuck process the spinlock was held (0) and no other thread was left to release it. Writing the unlocked value (-1) back from gdb let every stuck process finish exiting.

The lock is left held because of how the process exits. ExitProcess ends every other thread wherever it is and then runs the modules' detach routines. Driver::stop calls CloseRegistrationPreparation(), which reached the preparation pool through RegistrationPreparePool::Instance(). In a process that never prepared a shader, this created the pool during shutdown and started hardware_concurrency() / 2 threads (at least 2) only to close it. The first of the new threads to wait on the pool's condition variable initializes it under cond_locked. If the exit ends that thread at that point, the spinlock is never released. Without these threads, no run got stuck (see Tested).

This PR makes the pool start its threads at its first submission instead of in its constructor. Closing a pool that never ran a job then starts no threads. A submission after the pool is closed still runs on the caller's thread, which agc_driver_registration_prepare_shutdown checks. #4152 (d809e6d) kept shutdown from creating the prebuild pool by closing it only if it exists. The preparation pool now has Existing(), but a slot alone would let a submission after shutdown start threads, so this PR starts the threads lazily instead.

The new agc_driver_registration_prepare_idle_shutdown constructs the driver, records the process's threads, runs the libc shutdown and requires that no new thread is still running. On Windows it ignores threads whose start address is in ntdll.dll. Those are Windows thread-pool workers, which Windows starts on its own (a work item submitted with TrySubmitThreadpoolCallback runs on a thread with the same start address). The test fails on main (Windows and Linux) and passes with this PR. The Windows job builds only the execution tests listed in ANYPS5_AGC_EXECUTION_TESTS, so the second commit adds the new test to that list. Like agc_driver_registration_prepare_shutdown, it needs no Vulkan device.

docs/dev/ARCHITECTURE.md now says when the pool starts its threads.

This PR does not change the broader hazard of detach-time destructors running after ExitProcess has ended the other threads; #2612 handled that for _Exit. #5007 changes the same class (Submit, Main and the members), so whichever lands second needs a small rebase.

Tested

Native Windows 11 build (WinLibs GCC 15.2.0, Release) on an RTX 5080 (NVIDIA 616.92). No title.

A harness ran each test binary repeatedly with its ctest environment, 16 runs at a time. The harness and the tests were pinned to logical CPUs 8 to 11 to load them like a 4-vCPU runner. Any run that had not exited after 15 s got taskkill /F, and if it was still alive 5 s later, gdb. The main build is 0f2683c with only the new test added. main and this PR ran in alternating blocks of 400 runs (200 for the new test).

Test main 0f2683c this PR
agc_driver_wait_shutdown 49 of 1600 runs stuck 0 of 1600
agc_driver_resolution_shutdown 22 of 1600 stuck 0 of 1600
agc_driver_registration_prepare_idle_shutdown failed 789 of 800, the other 11 stuck passed 800 of 800
  • All 82 stuck runs had the stack above, with cond_locked at 0, were still alive 5 s after taskkill /F, and exited once cond_locked was set to -1. In a separate probe, taskkill /F reported for each of 11 stuck runs that the process could not be terminated, and each used 2.0 to 2.9 s of CPU time in 3 s while up to six of them shared the 4 CPUs.
  • One run at a time on the same CPUs, main got stuck in 6 of 200 runs of agc_driver_wait_shutdown and 8 of 200 of agc_driver_resolution_shutdown. This PR got stuck in 0 of 200 of each.
  • With this PR, a build of the new test that does not ignore ntdll.dll threads failed 132 of 800 runs under the same load. The test as committed passed 1600 of 1600 runs (the 800 above and 800 more).
  • A stress program (not added) started from a fresh pool in each of 1000 runs: 8 threads submitted 40 jobs each while another thread closed the pool after a random 0 to 400 µs. In every run all 320 jobs ran, no queued job started after the close returned, and a submission after the close ran on the caller's thread. Runs covered a close before any submission (20), after all of them (2) and in between (978).
  • Linux (WSL2 Ubuntu 26.04, GCC 15.2, CI configuration with lavapipe; only these tests and the libraries were built): the new test fails on main (50 of 50 runs) and passes with this PR (200 of 200 repeats). agc_driver_wait_shutdown, agc_driver_resolution_shutdown, agc_driver_registration_prepare_shutdown and agc_driver_prepared_publication pass.
  • Native full ctest of this branch (ANYPS5_REQUIRE_VULKAN=1, --parallel 4, two rounds): 1070 tests, 0 failed. Both rounds skip the same 7 tests, which tools/check_skips.py --job windows accepts, and tools/check_test_registration.py passes.
  • A CI-style configure with the new ANYPS5_AGC_EXECUTION_TESTS list builds agc_driver_registration_prepare_idle_shutdown from its source instead of the skip stub.
  • The branch merges cleanly into current main e152ba8, and tools/check_conventions.py passes.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: none
  • GPUs tested: NVIDIA GeForce RTX 5080, driver 616.92, Windows 11; lavapipe (Mesa 26.0.8, llvmpipe) on WSL2 Ubuntu 26.04
  • AI-assisted: yes. Written with Claude Code; I reviewed every line and ran the tests above.

🤖 Generated with Claude Code

nikowolf123 and others added 2 commits October 13, 2026 16:21
Driver::stop calls CloseRegistrationPreparation at every shutdown, and it
reached the pool through RegistrationPreparePool::Instance(). A process
that never prepared a shader therefore created the pool during shutdown
and started hardware_concurrency() / 2 threads (at least 2) only to close
it. On Windows the exit that follows ends those threads wherever they are.
When it ends one that holds winpthreads' global cond_locked spinlock,
which guards the first use of a statically initialized condition variable,
the Driver singleton's destructor spins forever in pthread_cond_destroy on
a condition variable it never used. The process keeps a core busy and
cannot be terminated, and ctest reports agc_driver_resolution_shutdown or
agc_driver_wait_shutdown as timed out.

The pool now starts its threads at its first submission, so closing a pool
that never ran a job starts none. A submission after the pool is closed
still runs on the caller's thread.

agc_driver_registration_prepare_idle_shutdown checks that a shutdown
without preparations leaves no thread running that it started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t a device

The Windows job builds only the execution tests named in
ANYPS5_AGC_EXECUTION_TESTS, so agc_driver_registration_prepare_idle_shutdown
only reported a skip there. The hang it guards against happens on Windows,
and the test needs no Vulkan device, so the job now builds it next to
agc_driver_registration_prepare_shutdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@oneandonlydean

Copy link
Copy Markdown
Collaborator

Closing: no title is known to reach this. With the queue at 700+ open PRs, please hold off on new PRs that don't name a title needing the change until it's down. If a title stops on this on current main, reopen with the title ID and the log line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* ci .github docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants