Skip to content

feat(shader): publish and load prebuilt resource plans - #62

Draft
JJenkx wants to merge 7 commits into
mainfrom
feat/prebuilt-resource-plans
Draft

JJenkx wants to merge 7 commits into
mainfrom
feat/prebuilt-resource-plans

Conversation

@JJenkx

@JJenkx JJenkx commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Publish existing resource plans alongside compiled variants when packing a populated renderer cache, then load those plans before the writable cache. Packed and loose plan entries require matching source, format and request keys and valid checksums. Missing or invalid entries fall back to normal compilation. Raw archive extraction does not synthesize runtime resource plans.

Seven linear topic/dependency commits on main 8856f180ca0b5605574842666ffbc8b0bb69cc4d; head 5fb29c2dca3e3f27e8f2b3eed7eb53bcf97d6345. Main's shader cache format 36 and the archive tool's 48-byte encoded-entry header are retained. The loader from boykopovar#5792 is already merged. No obsolete loader switches are restored.

This fork draft includes the earlier four archive-tool/dependency commits from SaloQT's boykopovar#6032, preserving authorship, followed by the resource-plan implementation, portable link correction and regression. boykopovar#6032 is now closed in favor of the split recompiler PR boykopovar#6296. This branch has not yet incorporated that revised split or the later tool trimming; dependency alignment remains pending before upstream submission. It is a development integration, not a duplicate upstream archive-tool submission.

Validation on this head: Linux x86-64 GCC Release, renderer SPIRV-Tools enabled. Full runtime libs, agc_shader_archive, shader_archive_tests, agc_shader_disk_cache_tests and shader_cache_file_read_tests built successfully. shader_cache_file_read, agc_shader_disk_cache and shader_archive passed with zero skips (2.28 s), with APS5_VERIFY_SRT_COMPACT=1. Conventions and whitespace pass.

The production cross-process regression removes writable plan entries before packed-plan and loose-plan loads. All 16 fixtures must produce their expected output without frontend builds; writable plans are restored for corrupt-pack fallback. The archive regression covers mixed entry types, mapping lifetime, plan serialization and rejected invalid/foreign entries. An earlier disabled-prebuilt-lookup control failed with 16 frontend builds / 16 plan misses; that control was not repeated for this base refresh.

The local build differs from the public head only by a private SPIRV-Tools teardown repair in the submodule. That dependency change is excluded from this PR; the official pin is not independently validated by these results. No current Windows, complete-suite, native title or FPS result is claimed. Earlier CI/runtime measurements apply only to their recorded revisions and are not validation of this head. No game assets, packs, raw logs or binaries are committed.

@github-actions github-actions Bot added agc libSceAgc* shader core/shader build CMake, submodules, 3rdparty docs Documentation needs rebase Conflicts with main, rebase needed labels Oct 14, 2026
@JJenkx
JJenkx force-pushed the feat/prebuilt-resource-plans branch from ce37234 to 060ec3d Compare October 14, 2026 20:32
@github-actions github-actions Bot added needs rebase Conflicts with main, rebase needed and removed needs rebase Conflicts with main, rebase needed labels Oct 14, 2026
@JJenkx
JJenkx force-pushed the feat/prebuilt-resource-plans branch from 060ec3d to ea92af9 Compare October 14, 2026 22:46
SaloQT and others added 7 commits October 15, 2026 06:57
CompileShaderUncached compiles one request without the process-lifetime
source cache or a resource snapshot, so batch tools can compile shader
archives with bounded worker lifetimes. Optional frontend and backend
timings report where each compilation spent its time.
EntryKey returns the validated key stored in an encoded cache entry
and IsCurrentEntryHeader checks that a file header belongs to this
compiler source and format, so offline tooling can group and screen
existing cache files without decoding them. Both validate against the
aligned payload layout.
…lt packs

agc_shader_archive extracts native shader programs from IoStore
containers, builds static AGC requests, compiles them through the
production recompiler with bounded workers, validates the SPIR-V, can
compile pipelines on a Vulkan device, and publishes the results as
prebuilt packs for the runtime loader. Captured requests can be
replayed offline with their actual targets, and probe reports exact
pack hit coverage and load timings.

Oodle Mermaid streams are decoded by a new decoder in Oodle.cpp whose
bitstream layout follows the MIT-licensed oozextract 0.5.5 decoder;
unsupported codecs and entropy forms throw and are recorded in
technical debt. The target is excluded from the default build.

Extraction of one measured title reproduces all 4,803 shader groups
byte-for-byte. Replaying eleven captured requests publishes an
eleven-entry pack, and probe then loads eleven of eleven from the pack
with an empty writable cache.

Depends on the uncached compile entry point and the prebuilt pack
loader commits.
New Markdown outside docs/ is rejected by the conventions check, so the
tool guide lives with the other developer documents.
@JJenkx
JJenkx force-pushed the feat/prebuilt-resource-plans branch from ea92af9 to 5fb29c2 Compare October 15, 2026 10:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* build CMake, submodules, 3rdparty docs Documentation needs rebase Conflicts with main, rebase needed shader core/shader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants