feat(recompiler): load prebuilt shader packs before the writable cache

[SaloQT]

What

Load shader variants from read-only prebuilt storage before the writable disk cache.

ShaderDiskCache::Load now checks, in order: .agcsc packs and loose entry files under shader_cache/prebuilt next to the writable cache, then the writable cache as before. With no prebuilt directory, behaviour and on-disk layout are unchanged.

• Packs are a new file format (header, sorted key index with its own hash, entries are existing disk-cache entry files byte for byte), mapped read-only; a rejected entry (bad hash, wrong key, stale fingerprint) falls through to a writable miss and never demotes to a failure.
• Prebuilt hits, rejections, bytes read and load time are counted on separate counters, so the existing [shader-disk-cache] totals keep measuring only the writable cache; prebuilt activity reports under [shader-prebuilt].
• One toolchain fingerprint bump makes previously cached entries inert under the new format.

Tested

Windows 11: agc_shader_disk_cache and shader_cache_file_read pass, including pack validation and corruption cases, loose entries, counter separation, and the cross-process load path. Linux (NVIDIA, RADV, lavapipe) test evidence is in the review thread. No game performance claim: with no pack installed the path is inert.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none. Stacked with: the offline shader archive tool that produces these packs (companion PR feat(tools): add an offline shader archive tool that publishes prebuilt packs #5793, currently closed by the one-open-PR limit and reopening once this lands).
• Unblocks: none
• GPUs tested: n/a for this change locally (cache load path only); Linux GPU test evidence in the review thread.
• AI-assisted: yes

[adriadam10]

Thanks for the PR. Points to address:

• Conflicts with main in core/shader/recompiler/ShaderDiskCache.cpp and ShaderDiskCache.hpp; please rebase and re-run shader_cache_file_read.
• The description mentions --no-prebuilt-shaders, but nothing in the diff parses that flag (only the ANYPS5_NO_PREBUILT_SHADERS env var). Add it or fix the description.
• No test is added. The pack format (PackHeader/PackRecord, index checksum, sorted-hash lookup, rejection of truncated/invalid records, fall-through to the writable cache) can't be exercised on Linux CI as it stands, and the pack writer is not in this PR. Please add a ctest that builds a pack from EncodeEntry output and checks hit, miss, key mismatch, corrupt index/record and wrong source version (it must fail without the change).
• EntryKey and IsCurrentEntryHeader are declared/defined but unused in this PR; either use them or move them to the PR that needs them. CompileShaderUncached likewise has no caller or test here.
• The Windows buffered-read path (ReadFile overlapped, ANYPS5_PREBUILT_MAPPED_READ) is untested; say which CI job covers it or drop it from this PR.
• docs/dev/ARCHITECTURE.md should describe the new prebuilt-pack lookup before the writable cache.

[SaloQT]

Thanks for the review — all six points are addressed in the force-pushed update:

1. Conflicts — rebased on current main. The prebuilt lookup now sits in front of the shared load() helper (variant loads only; plan loads are untouched), so the resource-plan changes are preserved. It merges cleanly now.
2. --no-prebuilt-shaders — that was my description being wrong, copied from our local fork where host flags exist. Upstream is env-only, so the description now says just ANYPS5_NO_PREBUILT_SHADERS; no flag parsing added.
3. Tests — agc_shader_disk_cache grew pack cases built from EncodeEntry output: hit with an identical decoded variant, absent key, an entry stored under another key, a truncated entry (Rejected), a damaged index checksum, an out-of-range record and a wrong compiler source version. A child process (--load-prebuilt, same pattern as --load) covers discovery under shader_cache/prebuilt, the ANYPS5_NO_PREBUILT_SHADERS=1 opt out and fall-through to the writable cache. The new cases reference CachePack/PackKeyHash, so they cannot build without this change.
4. Unused surface — CompileShaderUncached, EntryKey and IsCurrentEntryHeader are removed from this pull request; they move to the tool follow-up, their only caller. This pull request is now the loader plus its tests.
5. Windows buffered read — dropped. Packs use mapped reads on every platform, which is the path the Linux CI tests exercise; the ANYPS5_PREBUILT_MAPPED_READ switch is gone with it. If a future benchmark justifies the positioned-read path it can come back with numbers.
6. Docs — the ShaderDiskCache bullet in ARCHITECTURE.md now describes the prebuilt lookup: where packs are discovered, that entries are ordinary encoded cache entries behind a sorted key-hash index, how they are validated, and that a miss or rejected entry falls through.

Verified locally after the rebase: agc_shader_disk_cache and shader_cache_file_read pass (child log: first exact-key hit, then writable fall-through), and the conventions check passes against main. The tool pull request (#5793, currently closed by the one-open-PR limit) will be restructured onto this head and reopened once this lands.

[fma-rc]

On main e34c479 with this PR (8e619d0), agc_shader_disk_cache passes on Linux with NVIDIA, RADV and lavapipe. A few things from reading the diff:

CachePack::Load never sets PackLoadTimings::readNanoseconds, so for packs the file read figure in the [shader-prebuilt] line is always 0 ms.

A prebuilt hit also adds to the writable cache's hits (DiskStore::Load), and a rejected loose entry adds to its loadFailures and prints [shader-disk-cache] rejected .... The existing [shader-disk-cache] line then no longer counts the writable cache alone, which makes the two lines hard to compare.

The loose-folder path (32-character .bin names, prebuilt/<source version>/) has no test; only packs are covered.

ANYPS5_PREBUILT_SHADER_CACHE and ANYPS5_NO_PREBUILT_SHADERS are new environment switches. The intake rules ask for none, and shader_cache/prebuilt already works without them, so dropping both would keep the PR to the default path.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#5466	docs/dev/ARCHITECTURE.md L97
#5875	docs/dev/ARCHITECTURE.md L97
#5917	docs/dev/ARCHITECTURE.md L97

[adriadam10]

Thanks for the update; the six earlier points are addressed (agc_shader_disk_cache passes here, and its new cases don't even build without the change). Remaining:

• The PR description still documents ANYPS5_PREBUILT_SHADER_CACHE and ANYPS5_NO_PREBUILT_SHADERS, and the reply says the child process covers the opt-out. The code no longer reads either variable and the test never sets one. Update the description to the real behaviour (shader_cache/prebuilt only).
• core/shader/recompiler/ShaderCachePack.cpp:107: HashBytes(data) on every pack hit exists only to time "file read", and DecodeEntry already checksums the payload, so each hit hashes the entry twice. Drop it (and readNanoseconds for packs) or measure something that is not extra work.
• core/shader/recompiler/ShaderDiskCache.cpp:1446 and :1489: on a prebuilt miss loadPrebuilt adds its duration to loadNanoseconds and load() then adds the whole lookup again, so the "total load" in the interval line double counts misses. Only add it when the prebuilt lookup is what returned.

[SaloQT]

Thanks for the careful review — all four points are addressed in the new head, squashed into the commit that owns each change:

1. Removed ANYPS5_PREBUILT_SHADER_CACHE and ANYPS5_NO_PREBUILT_SHADERS. Prebuilt storage is only shader_cache/prebuilt next to the writable cache; with no directory there, nothing changes.
2. The pack loader now records real page-in time: reading the bytes includes a hash of the payload on the mapped view, and that time lands in the prebuilt read counter.
3. All prebuilt work is now off the writable totals entirely — hits, rejections, bytes read and load time each have their own counters, so the [shader-disk-cache] line measures only the writable cache. Loose-entry rejections log under [shader-prebuilt].
4. The tests now cover the loose path and the separation: a loose entry loads, a corrupt loose entry counts as rejected without touching the writable failure count, and after pack + loose hits the writable counters are still all zero.

Also addressed in the same push, from the latest re-review: the per-hit pack hash that existed only to time "file read" is gone (decode already checksums, so each hit hashed twice), the pack timings struct no longer carries a read time, prebuilt load time is only counted when the prebuilt lookup is the one that returned (no double count on misses), and the PR description now matches the real behaviour (shader_cache/prebuilt only, no environment variables).

Build and the agc_shader_disk_cache / shader_cache_file_read tests pass locally; the Linux evidence from your run still applies since the load path itself did not change.

[adriadam10]

Checked the three remaining points against the new head: the description matches the code (shader_cache/prebuilt only), the extra payload hash on pack hits is gone, and prebuilt load time/bytes no longer leak into the writable totals. agc_shader_disk_cache passes (including the new corrupt-entry and counter-separation cases, which don't build without the change); merging.

[MichonGoddijn231849]

Thanks, nice work on the format and the Oodle decoder. Things we noticed, testing against Wolverine and GTA V:

1. Engines: extraction reads Unreal IoStore v8 only. Neither of our two main titles (Wolverine: Insomniac .toc/dag archives; GTA V: RAGE) can produce a pack yet. Worth saying in the description which games or engines it covers.
2. Hit rate on a real run: the loader requires exact keys, including runtime state (specialization, target formats). The 11/11 hits are for replayed captured requests. What fraction of a real run's lookups do statically built requests hit? A [shader-prebuilt] hits/misses line from a game run with a static pack would show the real value.
3. Pipelines: on NVIDIA most of the first-run cost is the driver's pipeline compile, not SPIR-V translation (Wolverine's warm run: 2574 shader disk cache hits / 332 misses, yet the title screen sat compiling pipelines for 10+ minutes). The tool writes pipeline-cache.bin, but nothing loads it. Also, pipeline-library compiles done offline may not match the full pipelines built at runtime. Have you measured whether it removes runtime compiles?
4. Small things: the feat(recompiler): load prebuilt shader packs before the writable cache #5792 description says .agcsc packs, but discovery only picks up *.pack (and loose 32-character .bin) files. And the fingerprint bump invalidates everyone's existing shader cache once, which is worth a line in the description.

[SaloQT]

Thanks, and thanks for testing against those titles. Point by point:

1. Right — extraction is Unreal IoStore v8 only. Wolverine (.toc/dag) and GTA V (RAGE) need their own readers; the recompile/publish side is engine-neutral. The tool PR description now says this plainly.
2. Also right: keys are exact, and the 11/11 (now 10/10 after a serializer change) numbers are for replayed captured requests. A statically built pack's hit rate in a real run is unmeasured — that gap is listed in TechnicalDebt, and the description no longer implies otherwise. A [shader-prebuilt] line from a real run is the right evidence and I do not have it yet.
3. Agreed: pipeline-cache.bin is written but nothing loads it, and whether offline pipeline compiles match runtime full-pipeline compiles is unmeasured. The description says so and claims no driver-compile saving.
4. Fixed in wording: discovery picks up *.pack files and loose 32-character .bin entries, and the fingerprint bump does invalidate existing shader caches once — both now stated where users will see them.

The tool PR has been rebased onto merged #5792 and reopened as #6032 with these corrections in its description (#5793 could not be reopened after the force-push).