perf(shader): bound masked-lane work per shader

[MarcZX10]

What

Fixes #3504. Stray (PPSA02101) takes 146.7 seconds to prepare shader 0x200f780000 on current main, with nearly all the time in MaskedSelectEliminator::Eliminate. In MaskedSelectEliminator.cpp, MaskedLaneWalk::UnobservedWhereMasked resets its full program-scaled work limit for every candidate select. The issue's counters show more than 1.2 billion values visited across those repeated walks.

Keep the per-walk limit, and add one shared work budget for the entire shader. Charge the budget on both successful and early-failed walks. Once the budget is consumed, later proofs return “not proven”; a select is removed only when the existing proof completes successfully. The pass can leave more selects in the IR, but does not change shader semantics. The result reports the analysis work and budget for regression coverage.

The agc_masked_select regressions build a long chain of predicated selects, verify the analysis stops at its per-shader budget and keeps unresolved selects, and verify that a later select is still removed after an earlier masked proof spends part of the budget and conservatively fails.

#3391 changes the same pass to prove whole-quad mask implications for GTA V. Its behavior is separate from this per-shader work budget; this PR does not depend on it. The overlap is limited to the shared optimizer and test files, and may require a conflict resolution if either branch merges first.

I could not replay Stray's dumped shaders locally. The reported 146.7-second main result and counters are from the issue report. I asked the issue reporter to compare the two shader requests at 1x, 8x and 32x the per-walk limit, including SPIR-V validity; those title-specific measurements and budget tuning remain pending.

Tested

• Linux, GCC 12.2, Release: built agc_masked_select_tests.
• Focused CTest agc_masked_select — passed three consecutive runs.
• python3 tools/check_conventions.py --base upstream/main — passed.
• git diff --check — passed.

Checklist

• Based on current main; no open PR implements this per-shader budget behavior
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw; silent stubs are listed in TechnicalDebt
• No notes, investigation files or images added to the repository
• New third-party code is a submodule built from source
• Generic behavior, not specific to one title
• Depends on: none
• AI-assisted: yes

[adriadam10]

Thanks for the quick patch. The approach (a shared per-shader budget, charged by failed and successful walks) looks reasonable and the pass stays semantically safe, but the PR doesn't show yet that it fixes #3504:

• No before/after numbers: 0x200f780000 / 0x200f7a0000 timings on this head are needed (the issue shows 146.7 s / 406 s on main). Please post them from agc_shader_replay with the counters (walks, values visited, selects removed), or ask @ThatsLeo to run them in #3504.
• The budget is 32 × walkLimit, picked without data. Please show how the time and the number of removed selects vary with it on those shaders (the issue reports 4,616 removed after fda4aa9 vs 1,391 before), and that the SPIR-V stays valid. A budget that is spent by the first masks leaves later selects in, so report that cost too.
• agc_masked_select only checks that the budget gets consumed. It can't fail for the wrong reason (it doesn't compile without the change). A case where a select provable within the budget is still removed after earlier walks have spent part of it would pin the "proof completes, select removed" half.

Happy to merge once the measurements are in.

[MarcZX10]

Added commit b58c0428 with the requested follow-up regression. Its first masked proof visits a chain, then conservatively fails when the value reaches an image write observed under a wider exec mask. A later proof under the narrower mask still completes and removes its select. The test also checks that some, but not all, of the shader-wide budget was spent. agc_masked_select passes three consecutive Release runs on Linux/GCC 12.2.

The two Stray replay request files are not available in this workspace, so I can't provide the timing, select-count or SPIR-V results from here. I asked the issue reporter to run both shaders at 1x, 8x and 32x the per-walk limit, and included the requested output fields in the issue comment. The PR description records this as pending rather than claiming an unverified improvement.

[adriadam10]

Thanks for the follow-up test; agc_masked_select passes here on the PR merged onto current main (Linux, scoped build, 404/404 ctest in libSceAgc/libSceAgcDriver), and the new stats make the tests fail to build without the source change.

What is still missing is the one measurement the claim rests on, which we can't produce without the Stray requests:

• 0x200f780000 / 0x200f7a0000 replayed on this head vs main (time, values visited, selects removed, SPIR-V valid), as requested in #3504. If @ThatsLeo posts it there, it is enough.
• The 32x multiple needs that data (or a stated reason), since a lower value than the shader needs leaves selects that other shaders (e.g. the ones in #3391) rely on being removed.

Once the numbers are in, this is ready to merge from our side.

[oneandonlydean]

Conflicts with main in MaskedSelectEliminator.cpp and tests/MaskedSelect.cpp since #3391; please rebase.

ThatsLeo's tables in #3504 cover the numbers (141 s → 1.3 s and 385 s → 2.4 s at 32x, SPIR-V valid). They also show that with this PR alone 0x2013010000 stops compiling below 128x:

packed pixel ancillary input has an unsupported live use (used by SelectU32 ? ?)

So either this goes in after #4031 (list it under Depends on), or the multiple has to be one that keeps that shader compiling.

[blawesom]

Rebase handoff — this branch was ~800 commits behind main and conflicting. I rebased it onto current main and verified the result; the authorship of both commits is unchanged.

The conflicts were semantic, not mechanical — main evolved MaskedLaneWalk (the guarded first-lane reads and whole-quad work landed in the meantime), while this PR adds the shader-wide work budget. The rebased merge keeps both: main's nesting/uniform-aware constructor and sameIteration handling, plus this PR's remaining/allowance/ChargeWork budget accounting. The test file keeps main's new whole-quad tests and this PR's budget tests together.

Verified on Linux (CachyOS, kernel 7.2.9, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, AMD RX 9070 XT / RADV):

• builds; tools/check_conventions.py --base origin/main passes
• full suite: 949/949 passed — including the PR's own budget tests (masked-lane implication work stays within a shader-wide budget, a later bounded masked-lane proof can still remove its select) and main's whole-quad masked-select tests

To pick it up (either works):

git fetch https://github.com/blawesom/AnyPS5 rebase/3550
git checkout 3550  # your branch
git merge --ff-only FETCH_HEAD   # or reset to it

Or cherry-pick the resolution from blawesom:rebase/3550 (head 2289cbeac). Happy to adjust the resolution if you'd merge the two mechanisms differently.

[MarcZX10]

Closing this as superseded by merged #4331. Its backward-pass masked-lane analysis fixes the repeated per-select walk that motivated the shared budget and avoids the budget-dependent selects noted in #3504. The budget implementation is no longer needed on current main; #4331 documents the replay evidence and full-output behavior.