feat(tools): add an offline shader archive tool that publishes prebuilt packs

[SaloQT]

What

An offline tool that extracts shader programs from a game's archives, recompiles them ahead of time, and publishes the results as a prebuilt shader pack for the loader in #5792 (merged).

• tools/shader-archive (new): reads Unreal IoStore v8 .utoc/.ucas pairs, decodes the Oodle Mermaid quanta those archives use with its own clean-room decoder (no vendored Oodle), scans shader libraries, and builds shader variants fully offline through the existing recompiler. Output is published atomically as a *.pack file (the format feat(recompiler): load prebuilt shader packs before the writable cache #5792 loads) plus a manifest.
• Two small recompiler commits first: an uncached compile entry point so tools do not write through the disk cache, and encoded entry-key helpers so the tool builds entries in the cache's exact on-disk format.
• Remaining decoder limits (other Oodle codecs, other entropy modes, checksums) throw typed errors and are listed in docs/dev/TechnicalDebt.md; the tool fails on archives it cannot decode instead of guessing.

Scope, honestly: extraction covers Unreal IoStore v8 only today; other engines' archives (Insomniac, RAGE) need their own readers. Keys are exact, so statically built variants only hit when the runtime request matches exactly — the verified end-to-end numbers are for replayed captured requests, and a real-game hit-rate measurement with a statically built pack is still open (also in TechnicalDebt). The tool also writes pipeline-cache.bin, but nothing loads it at runtime yet; whether offline pipeline compiles remove driver compile cost is unmeasured, so this PR claims no pipeline-compile saving.

Tested

Windows 11: rebuilt on current main; shader_archive, agc_shader_disk_cache and shader_cache_file_read pass. Full real-archive run: 4,803/4,803 IoStore groups decoded byte-identical to the licensed-SDK reference, 80,794 shader hashes matching. Offline build + publish + probe against captured requests: 10/10 exact-key hits. Not tested on Linux locally (no Linux box here); CI covers it.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: feat(recompiler): add an uncached compile entry point and entry key helpers for offline tools #6296
• Unblocks: none
• GPUs tested: offline CPU recompile only; n/a
• AI-assisted: yes

[adriadam10]

Thanks for the work. I'm requesting changes because, as the description says, the runtime benefit hasn't been shown yet, and the PR is too big to merge on trust.

1. Measure the hit rate with a static pack. The 10/10 hits come from replaying requests captured from the emulator, which is just a copy of an existing cache. What this tool claims is that variants built from the archive hit live requests, and that isn't measured. Keys include target, binding layout and stage configuration, so guessed variants can miss every time. Please post [shader-prebuilt] hits/misses from a real run with a pack built by build --paks only, along with load time with and without the pack. Without that number we can't merge 4k lines.
2. Say what it saves. Since #639 the RDNA → SPIR-V translation already runs at registration. First-draw cost is specialization plus the driver pipeline compile, and this PR doesn't touch either (pipeline-cache.bin isn't loaded). Please state the expected saving at load time, measured.
3. Split it. The two recompiler commits (CompileShaderUncached, the encoded entry-key helpers) go in their own PR, with tests. The tool goes in a follow-up that depends on it. Mesh shader-object binaries, pipeline-cache.bin and the APS5_PIPELINE_SALT benchmarking path aren't used by the emulator, so leave them out until something consumes them.
4. Oodle decoder provenance. The description says groups were verified byte-identical "to the licensed-SDK reference", while TechnicalDebt says "against the previous decoder". Which one was it? Please clarify the provenance in the PR. A decoder for a proprietary codec that was checked against the licensed SDK needs a clear statement that no SDK code or headers were used.
5. Docs reference things that aren't in the repo. docs/dev/ShaderArchive.md mentions "the Rust extractor", its shaders.jsonl/groups/*.bin and "the Rust tool's static-target profile". None of these are in the tree. Please drop them or point to what users are supposed to run.
6. Linux. Only tested on Windows. CI doesn't run the tool on a real archive, so please run shader_archive and one extract/build on Linux, or say it's untested there in TechnicalDebt.
7. Conflicts. ShaderDiskCache.hpp overlaps with #5509 and #6243 (see the overlap comment). Rebase once those settle.

[SaloQT]

Thanks for the thorough review. This push covers the mechanical points; the measurement and the split are the next round and I am not claiming they are done.

Done in this push:

• Docs no longer reference things outside the tree (the Rust extractor, its shaders.jsonl/groups/*.bin, and its static-target profile are gone from docs/dev/ShaderArchive.md).
• Removed the outputs nothing consumes: pipeline-cache.bin writing, the APS5_PIPELINE_SALT benchmarking path, and the mesh shader-object creation/binary dump. GPU validation now only builds pipelines.
• Provenance clarified in TechnicalDebt: the decoder is new code written from the MIT-licensed oozextract format reference, uses no SDK code or headers, and its output was checked byte-for-byte against the team's earlier reference decoder output (not against SDK code at runtime). The earlier description wording was sloppy; sorry.
• TechnicalDebt now states the tool is verified on Windows only and untested on Linux.
• Rebased onto current main, so the ShaderDiskCache.hpp overlap is resolved for now; I will rebase again if fix(shader): read cube T#s through DIM 2D image_sample and image_load as face 0 #5509/perf(shader): select the record of a T# whose offset the CPU evaluates #6243 move it.

Still open, in your order:

1. Real-run hit rate with a pack built by build --paks only, plus load time with/without the pack — not measured yet; agreed this is the number that matters.
2. Measured saving statement — depends on (1); the description already claims no pipeline-compile saving.
3. Split: the two recompiler commits will move to their own PR with tests, and this tool becomes the follow-up. Because of the one-open-PR limit that means closing this PR and opening the recompiler one first; I will do that next so the small part can land on its own merits.

[SaloQT]

Split as requested: the two recompiler commits (uncached compile entry point, encoded entry-key helpers) now have their own PR with tests (linked below once opened). This tool PR will return as the follow-up on top of it, with the real-run hit-rate measurement. Closing this one because of the one-open-PR limit; the branch feat/shader-archive-tool keeps the trimmed tool at its current head.

[SaloQT]

The split recompiler PR is #6296.

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 674	CMakeLists.txt
PR 907	docs/dev/TechnicalDebt.md
PR 1368	docs/dev/TechnicalDebt.md
PR 1384	docs/dev/TechnicalDebt.md
PR 1395	docs/dev/TechnicalDebt.md
PR 1601	docs/dev/TechnicalDebt.md
PR 1835	CMakeLists.txt docs/dev/TechnicalDebt.md
PR 2780	docs/dev/TechnicalDebt.md
PR 3247	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 3786	CMakeLists.txt
PR 3986	docs/dev/TechnicalDebt.md
PR 3998	CMakeLists.txt
PR 4072	core/shader/recompiler/Recompiler.cpp
PR 4650	core/shader/recompiler/Recompiler.cpp
PR 5021	CMakeLists.txt core/shader/recompiler/Recompiler.cpp docs/dev/TechnicalDebt.md
PR 5500	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 5609	core/shader/recompiler/Optimization/include/Optimization/ResourceProgram.hpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 5656	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 5710	CMakeLists.txt
PR 5925	core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 5958	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 5976	CMakeLists.txt
PR 6039	docs/dev/TechnicalDebt.md
PR 6044	CMakeLists.txt
PR 6067	CMakeLists.txt
PR 6177	core/shader/recompiler/Recompiler.cpp
PR 6178	core/shader/recompiler/Optimization/include/Optimization/ResourceProgram.hpp core/shader/recompiler/Recompiler.cpp
PR 6222	core/shader/recompiler/Recompiler.cpp
PR 6232	core/shader/recompiler/ShaderDiskCache.cpp
PR 6268	CMakeLists.txt
PR 6280	core/shader/recompiler/ShaderDiskCache.cpp
PR 6412	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 6450	CMakeLists.txt
PR 6454	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 6470	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 6486	core/shader/recompiler/ShaderDiskCache.cpp
PR 6488	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 6503	core/shader/recompiler/Recompiler.cpp
PR 6539	core/shader/recompiler/Optimization/include/Optimization/ResourceProgram.hpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 6611	core/shader/recompiler/Recompiler.cpp
PR 6612	core/shader/recompiler/Recompiler.cpp
PR 6679	core/shader/recompiler/Recompiler.cpp
PR 6683	core/shader/recompiler/ShaderDiskCache.hpp
PR 6733	core/shader/recompiler/Recompiler.cpp
PR 6746	CMakeLists.txt
PR 6757	CMakeLists.txt
PR 6758	docs/dev/TechnicalDebt.md
PR 6762	core/shader/recompiler/Recompiler.cpp
PR 6858	core/shader/recompiler/Optimization/include/Optimization/ResourceProgram.hpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 6879	CMakeLists.txt core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 6924	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 6941	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 6946	CMakeLists.txt core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 6947	CMakeLists.txt core/shader/recompiler/Optimization/include/Optimization/ResourceProgram.hpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 6959	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 6970	core/shader/recompiler/ShaderDiskCache.cpp
PR 6977	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 7039	CMakeLists.txt
PR 7085	core/shader/recompiler/ShaderDiskCache.cpp
PR 7201	CMakeLists.txt core/shader/recompiler/Recompiler.cpp
PR 7223	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 7264	CMakeLists.txt
PR 7296	core/shader/recompiler/ShaderDiskCache.cpp
PR 7316	CMakeLists.txt
PR 7332	CMakeLists.txt core/shader/recompiler/Recompiler.cpp
PR 7381	core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 7390	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 7395	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp core/shader/recompiler/ShaderDiskCache.hpp
PR 7410	core/shader/recompiler/Recompiler.cpp
PR 7445	core/shader/recompiler/Recompiler.cpp
PR 7484	core/shader/recompiler/Recompiler.cpp
PR 7520	core/shader/recompiler/ShaderDiskCache.cpp
PR 7522	core/shader/recompiler/Recompiler.cpp
PR 7523	CMakeLists.txt

[adriadam10]

Thanks for the update. Checking this head against the points of the previous review:

• Hit rate not measured (previous point 1). The description still says the real-game hit-rate measurement with a statically built pack is open. The 10/10 hits are replayed captured requests, which don't show that keys built from the archive match live ones. Please post [shader-prebuilt] hits/misses from a run with a pack built by build --paks only, with load time with and without the pack.
• Saving not stated (point 2). Depends on the above. Without a measured gain, 4k lines of a proprietary-codec decoder plus extractor can't be merged on trust.
• Split only half done (point 3). The branch still carries b280f00899 and c5af57b355 (the uncached compile entry point and the entry-key helpers), which are what #6296 adds. Once #6296 lands, drop them here and rebase so this PR contains only the tool; until then it duplicates and will conflict with it.
• Linux untested (point 6). CI only builds the relinker for this PR, so shader_archive has not run on Linux; the Windows run is the author's. Please run shader_archive on Linux or keep the "untested on Linux" note in TechnicalDebt accurate.

Happy to review again once #6296 is merged and the hit-rate number is posted.