fix(agc): keep a shader library part's modules until the part is destroyed

[LeandroLP23]

Carrying @poonasor's #3149, which the intake check closed when it was opened. The change is theirs (8acd0113, cherry-picked onto current main with poonasor kept as the author; it applied without conflicts).

What

Pipeline destroys its shader modules right after LinkPipelineFromLibraries returns, but the pre-rasterization and fragment shader library parts built from them stay cached and are relinked later by the background VK_PIPELINE_CREATE_LINK_TIME_OPTIMIZATION_BIT_EXT link (#2713). On AMD's Windows driver that link faults inside amdvlk64.dll, at varying offsets, reading freed heap memory. Destroying a module after pipeline creation is valid Vulkan, so this looks like a driver defect; the change works around it.

When the optimized link is enabled, the library store takes over the modules of each newly built shader part and destroys them in ClearPipelineLibraries, after the pending links have finished. Modules of a pipeline whose parts all came from the cache are destroyed as before, and nothing changes with APS5_NO_PIPELINE_LTO=1. Cost: one retained VkShaderModule per stage of each unique shader part, for the device's lifetime.

This is the AMD half of #2777: #2869 fixed the destroyed layout, which did not change the AMD failures (as noted there and in #2755). Addresses the segfaults in #2944 and #2755.

How the cause was narrowed down (RX 6700 XT, on 57f96fd2 + 51078717, agc_driver_vertex_fetch and agc_driver_provoking_vertex):

• gdb puts the faulting thread in the LinkPipelineFromLibraries job, inside vkCreateGraphicsPipelines.
• APS5_NO_PIPELINE_LTO=1: pass.
• Optimized link run synchronously under the store lock instead of on the worker: still faults (2 of 3 runs each), so it is not a race in our code.
• Optimized link with a null VkPipelineCache: still faults 3/3.
• Shader modules never destroyed: pass 4/4 each (0/4 without).
• Implicit layers disabled (VK_LOADER_LAYERS_DISABLE=~implicit~): still faults.

Tested

@poonasor's run:

Windows 11 Pro 23H2 (22631), WinLibs GCC 15.2.0 r7, Ninja, Release, BUILD_TESTING=ON. AMD Radeon RX 6700 XT (Navi 22, 0x73df), Adrenalin 25.10.45.05 (driver 32.0.21045.5002), the only Vulkan device. %LOCALAPPDATA%\AMD\VkCache emptied before each agc_ run, ctest -R "^agc_" -j1:

Build	Failed of 324
main 379d8b4e	24: 22 SEGFAULT in amdvlk64.dll, plus sdwa_float_selectors and sdwa_ldexp_f16
main 379d8b4e + this PR	3: pixel_interlock, sdwa_float_selectors, sdwa_ldexp_f16, no SEGFAULT

The 22: agc_pipeline_cache, agc_pipeline_library, and agc_driver_ mesh, conservative_rasterization, depth_bias_per_face, dual_source_blend, exec_host_lanes, ngg_vertex_wave_info, packed_unorm_color_target, pixel_fixed_point_position, pixel_helper_lanes, pixel_interlock, pixel_lds, pixel_lds_lane_slots, pixel_uint16_export, pixel_valid_mask, pixel_wave_reduction, prepared_exports, provoking_vertex, scalar_swappc_fetch, srgb8_color_target, vertex_fetch.

Full parallel ctest with this PR (warm driver cache): 724 of 732 passed, 5 skipped (video_out flip tests, guest_video_out_wqhd_detection_no_vulkan), the same 3 failed.

The 3 remaining failures are wrong results, not faults, and are not addressed here: the two sdwa f16 ones are the values already in #2944; pixel_interlock no longer faults and fails with pixel interlock wave64 BDA fault: pixel 0 counted 1 of 0 overlapping read-modify-writes, which it also does with APS5_NO_PIPELINE_LTO=1.

Title: Dreaming Sarah (PPSA02929), which faulted in amdvlk64.dll+0x21eef34 at its first graphics submission on this machine (discussion #3018), relinked against this build runs for 90 s with no fault in the log. I did not check the rendered output.

Not tested on NVIDIA, Intel or Linux.

python tools/check_conventions.py --base origin/main passes.

Title confirmation from @poonasor (comment on #2944): Dreaming Sarah (PPSA02929), which faulted in amdvlk64.dll at its first graphics submission on the RX 6700 XT, reaches gameplay with this change and renders correctly at about 60 FPS, with audio and controls working and no fault in the log.

My run on RDNA 3.5: Radeon 860M (0x1114), AMD driver 32.0.13058.8001, Windows 11 (26300), WinLibs GCC 15.2.0 r7, Release, %LOCALAPPDATA%\AMD\VkCache and build/tests/shader_cache emptied first, ctest -R agc_ -j1 (also on #2944):

Build	Failed of 335
main 181420f	27, 13 of them access violations in amdvlk64.dll
main 181420f + this change	0, cold and warm

Fork CI on this branch (run): Windows and the three relinker jobs pass. Linux fails agc_driver_image_packed16_load and agc_driver_flat_apertures, which also fail with an unrelated change on the same main (run), and agc_unit_shadow ("the partial publish did not copy exactly the partly covered unit", the open #1429), which creates no pipelines.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes (Claude Code, for carrying and the RDNA 3.5 run; see fix(agc): keep a shader library part's modules until the part is destroyed #3149 for the author's disclosure)

[Dextroid17]

A workaround is only reviewable when it says what it is working around, and this one says it exactly: destroying a shader module after pipeline creation is valid Vulkan; the deferred VK_PIPELINE_CREATE_LINK_TIME_OPTIMIZATION_BIT_EXT link that the library parts stay cached for faults inside amdvlk64.dll at varying offsets, reading freed heap memory; and so it is a driver defect rather than a use-after-free on our side. Writing that down — driver, module, symptom, and the fact that the specification allows what we do — is what turns a workaround into a decision someone can revisit when the driver changes.

The ownership transfer is the right mechanism. Having the library store take over the modules of each newly built shader part and destroy them in ClearPipelineLibraries after the pending links have finished ties the lifetime to the thing that actually needs it rather than to the pipeline that created it. Doing that for every part built while the optimized link is enabled, rather than only for the ones that faulted, is the kind of difference that prevents a second bug report later.

One thing worth stating explicitly, since the fault belongs to one driver: the workaround is unconditional. There is no vendor or device check anywhere on this path — APS5_NO_PIPELINE_LTO=1 is the only way out of it. That is the right default for a fault seen on a single vendored driver, because the alternative is a title that keeps crashing on a driver nobody wrote a guard for; but it does mean the entry should say so plainly, so whoever meets the same fault elsewhere knows where to look first.

Written with AI assistance.

[Mouhand-Kaddo]

Windows title test on current main 974116d38: Demon's Souls PPSA01342 v1.04, Windows 11, RX 6900 XT, driver 32.0.21045.5002 (Vulkan reports 26.8.1), MinGW-w64 Release, default ANYPS5_ENABLE_SPIRV_TOOLS=OFF. The test stack also includes #4021, #4018, #637 and #4049 to reach this graphics stage.

After passing the earlier audio and resummarize stops, the game faults at amdvlk64.dll+0x9e251f, reading 0xffffffffffffffff. A separate run with APS5_NO_PIPELINE_LTO=1 produces the identical fault. No frame or menu is verified.

A private probe outside Git narrows this to the initial fragment-shader library creation, before that creation returns: VK_GRAPHICS_PIPELINE_LIBRARY_FRAGMENT_SHADER_BIT_EXT, one fragment stage, 61,693 SPIR-V words. The preceding pre-rasterization part completes. Khronos validation reports no errors before the fault, and the privately captured fragment passes standalone SPIRV-Tools validation for Vulkan 1.3.

This appears to be a separate initial library-compilation failure, beyond the retained-module lifetime handled here. I am reducing it further; I have not rerun the full AGC suite or established a driver-wide cause. Game data and generated captures stay local.

AI-assisted investigation and write-up; these are results from actual local Windows runs.