fix(shader): memory amplification in ShaderDiskCache and quadratic scans in GraphBuilder and InstructionTranslator

[silentone12725]

What

Three fixes in the shader recompiler:

ShaderDiskCache — memory amplification from wrong List minimum

The invocation decoder passed minimum=8 to reader.List for DescriptorBinding entries. The actual minimum serialised size is 114 bytes (5×uint32 + 11×uint64 list-count headers + 2×bool + 1×uint32 enum). A crafted cache file with a large entry count could cause the list allocator to reserve ~14× the expected memory before decoding a single entry. Fix: pass 114 as the minimum.

GraphBuilder — O(K×N) scan in resolveJumpTable / resolveDwordJumpTable

Both functions scanned all N program instructions to find any direct branch whose target falls in a given range, called once per indirect-branch block. With K indirect branches the total work was O(K×N). Fix: build a sorted std::vector of direct-branch targets once in Build (shared with splitIntoBlocks and linkBlocks, next to the swappc call analysis), then use std::upper_bound for an O(log N) check per call.

InstructionTranslator — O(N×M) loop via isCodeTableLoad

isCodeTableLoad called std::find over cfg.codeTableLoadProgramCounters (M entries) at every one of N instructions, in two loops, and the second loop searched cfg.codeTableLoads again for each hit. Fix: build an std::unordered_set and an std::unordered_map from the cfg data once before the loops, and remove the now-unused isCodeTableLoad.

Tested

• Linux: full build of shader_recompiler_agc_driver with submodules; agc_shader_control_flow_tests, agc_shader_memory_tests, agc_shader_fusion_tests and agc_shader_disk_cache_tests passed
• Conventions check: passed (python3 tools/check_conventions.py --base origin/main)
• The 114-byte minimum was checked against the encoder: 5 fixed 4-byte fields (20), guestDescriptor (8), two bools (2), the shape enum (4), nine flag lists (72) and imageSamplers (8) is 114.
• Pattern benchmarks (the old and new loop shapes in isolation, g++ -O2, best of 5; not a measurement of a whole shader compile): the GraphBuilder range check takes 8.2 ms against 0.11 ms for 100 000 instructions and 100 indirect jumps, and 77 ms against 0.16 ms at 1 000 jumps. The InstructionTranslator lookup takes 4.1 ms against 0.46 ms for 100 000 instructions and 256 table loads. With 16 table loads the set is no faster (0.28 ms against 0.34 ms), so the gain only shows on programs with many jump tables.
• Build workflow run on the fork, dispatched on head eb4bb29b: passed for Linux, Windows, and the relinker tests on Ubuntu (g++), macOS (clang++) and Windows (g++) (run 37835318482). It is not in the PR checks because the upstream run for this head is waiting for maintainer approval (CI: 38 open PR heads are queued at action_required, so their PRs show no checks #1631).

Changes since the first revision

• Rebased onto current main. Conflicts in GraphBuilder (the new swappc call analysis) and ShaderDiskCache (bindings removed from the artifact and allocation structs upstream) are resolved; the 114-byte minimum now applies to invocation.bindings, the only remaining caller.
• The two relinker commits this branch carried are removed; they belong to fix(relinker): reject zero section header entry size in ElfReader #1521 and perf(relinker): replace O(n²) zero-padding scan in StrictReachability #1522.
• The UpdateStamp early return is reverted: the error_code overload of last_write_time does not throw, and the early return would skip stale-cache and .tmp cleanup when the stamp cannot be written.
• isCodeTableLoad is removed, as nothing calls it any more.
• Added the pattern benchmarks and the 114-byte derivation (see Tested).

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw; silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes

[klosixgaming-arch]

Good performance and correctness improvements across three areas. The fixes are well-justified with clear explanations of the problems (memory amplification, O(n²) scans, out-of-bounds read).

Strengths:

• Proper use of sorted vectors and binary search for efficient lookups
• Correct fix for the memory amplification issue by using proper minimum sizes
• Good handling of edge cases in the ELF parsing fixes

Minor notes:

1. The PR is marked as draft with "needs rebase" label - appropriate given main has moved forward
2. Consider whether these three fixes should be separate PRs for easier review/merging, though bundling related performance fixes is also reasonable

This looks ready to merge once rebased and CI passes.

[klosixgaming-arch]

Excellent performance and correctness fixes across three areas of the shader recompiler. This PR addresses real issues that could impact both memory usage and compilation speed.

Strengths:

• ShaderDiskCache: Correctly identifies and fixes the memory amplification issue by passing the proper minimum size (114 bytes) to reader.List for DescriptorBinding entries
• GraphBuilder: Smart optimization replacing O(K×N) linear scans with sorted vector + binary search using std::upper_bound - this will significantly improve compilation time for shaders with many indirect branches
• InstructionTranslator: Good use of unordered_set and unordered_map to eliminate O(M×N) loop complexity in code table load detection

Technical notes:

• The approach of building the sorted direct branch targets vector once in Build() and passing it through is clean and efficient
• Proper handling of edge cases (empty programs, missing code table entries)
• Good documentation of the issues and solutions in the PR description

These optimizations will improve shader compilation performance without changing behavior. Ready to merge.

[silentone12725]

Hi @klosixgaming-arch, the branch has been rebased onto main and conventions checks pass. Would you be able to merge this when you get a chance? Thanks!

[oneandonlydean]

Closing: these are three unrelated changes, and none shows a need. The List minimum only matters for a crafted local cache file, and the speedups are measured on isolated loops rather than a shader compile (#218). If a title's shader compile is slow because of one of them, please reopen that one on its own with before/after numbers.