feat(agc): trace shader failures through draw submission batches

[sktharunbalaji13-cmd]

What

Add opt-in diagnostic correlation across draw attempts, shader selection and preparation failures, actual draw-command recording, and recorder batch submission in the AGC driver.

Each draw attempt receives a correlation token. Relevant failure and recording events preserve that identity. Recorder batches collect a deduplicated set of the tokens associated with commands recorded in the batch, allowing one batch to represent multiple draws. Submission-attempt diagnostics are distinguished from confirmation records emitted only after successful queue submission and finalized required recorder bookkeeping. Diagnostic formatting and logging are bounded and best-effort to prevent trace failures from masking original exceptions or blocking otherwise valid submission.

Tested

Clean PR branch \eat/shader-draw-submission-correlation-5166-fresh, rebased on current \main\ (\80249ea54).

• Windows local: CMake/Ninja configure succeeded; \core/libs/libs/libSceAgcDriver.prx\ built successfully; NID patching reported \OK.
• \ctest -R agc_capture_trace\ passed 8/8 tests.
• \ctest -R agc_driver_draw_shader_failure\ passed 6/6 tests.
• Total 14/14 tests in \libSceAgcDriver\ pass 100%.

Checklist

• Based on current \main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (\NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md\ files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none (diagnostics-only change; no title-specific trace evidence)
• GPUs tested: n/a; no GPU tests ran for this tracing-only change
• AI-assisted: yes

[MarcZX10]

I inspected the failing conventions run. It reports 39 errors: inline comments in CaptureTrace.hpp are disallowed unless the PR also documents the associated technical debt under docs/dev/TechnicalDebt.md or docs/dev/debt/, and the PR description is missing the required What, Tested, and Checklist sections. Please address those findings and rerun the check; the current PR body has validation details, but they need to be placed under the required headings so the repository check can recognize them.

I also compared this with #5238. Both PRs change Execution/src/Driver/Draw/Draw.cpp and the libSceAgcDriver CMake target, while tracing different points in the draw path. Could you clarify how the two tracing mechanisms are intended to compose and whether one can reuse the other, so they do not add duplicate or conflicting instrumentation?

[sktharunbalaji13-cmd]

Thank you for the review. The conventions findings are addressed in commit 6b5c7fbda03e9b9469e786b34d98a87dde438383: the non-technical-debt diagnostic comments were removed, the PR body now uses the required ### What, ### Tested, and ### Checklist sections, and tools/check_conventions.py passes locally against base dacd78525e537a779682032c0ace7f7af120af05. The GitHub conventions run subsequently passed.

On composition with #5238: they are intended to be complementary. PR #5238 gives per-draw shader address information (stage, role, code/entry address, queue, submission context), which helps identify the selected program. This PR assigns a correlation token to each draw attempt and tracks whether the draw reaches selection, actual command recording, submission attempt, and successful queue submission; it also associates recorded draws with a deduplicated set of tokens per recorder batch. In practice, #5238 helps answer "which shader was selected?" while this PR helps answer "did that draw attempt survive setup and get recorded/submitted?"

Both PRs touch the same Draw.cpp and CMake target, so their merge order will matter. If #5238 lands first, this PR should adapt to its address-formatting and per-draw structures rather than duplicating them. I avoided copying or renaming its address-formatting implementation; the remaining overlap is in the shared Draw/composition points where both may hook logging. They do not need to emit duplicate records for the same event, but some reconciliation at those shared points is likely during integration.

No runtime/game validation is claimed yet, and the platform build jobs remain skipped for this PR.

[MarcZX10]

Thanks, that addresses both points. The distinction between identifying the selected shader and tracking a draw attempt through recording/submission is clear, and your note about reconciling the shared Draw hooks if #5238 lands first gives a workable integration plan. I verified the new conventions check passes; the platform build jobs remain skipped for this path. No runtime validation is claimed.

[CnRJay]

Checked the cost with the trace off: the token is 0 and NoteDrawTraceToken returns at once, so the disabled path stays cheap. Two points about the enabled path and the scope:

• The dedup in NoteDrawTraceToken is quadratic per batch. It runs std::find over open->drawTraceTokens for every recorded draw. A batch holding a few thousand draws (common in a busy frame) then does millions of comparisons on the queue worker, under the recorder's open-batch state, while the trace measures that same worker. That distorts the timings the correlation is meant to explain. The tokens come from a counter, so a draw's commands are recorded contiguously and the token can only repeat back to back. Comparing against back() gives the same dedup in O(1). If tokens can interleave, an unordered_set beside the vector also works.
• The draw-path refactor would be better split out. Moving per-stage materialization into Driver::materializeDrawStage (27 parameters, several of them out-parameters) changes the hottest function in the driver as part of a diagnostics PR. It may well be behaviour-neutral, but reviewers have to verify that across ~200 lines, and it conflicts with every other open PR that touches Driver::draw (#5238 already). Keeping the token as one extra argument through the existing code, and leaving the restructure to its own refactor(agc) PR, would make both easier to review.

Minor: the catch (...) {} in NoteDrawTraceToken hides allocation failures. In a diagnostics path, losing a token silently makes the trace look complete when it isn't. Letting it throw, or counting the drops in the trace, is safer.

[sktharunbalaji13-cmd]

Thank you for the detailed review. I’ve addressed the three points in commit 2c3502d01f902b3cd6de2a67f658d68b35bf1389.

A. Token deduplication

NoteDrawTraceToken now uses an accompanying std::unordered_set beside the existing ordered token vector. A token is pushed only when the set insert succeeds, so dedup is O(1) average instead of O(batch size) per recorded draw, duplicates still cannot repeat in the vector, and ordering is unchanged. The side set is also rolled back if the vector append throws, so a failed append does not leave later recording unable to reintroduce that token.

B. Scope of the draw-path change

The PR mainly adds parameters rather than introducing a broad new draw refactor. The existing Driver::materializeDrawStage helper was already present upstream; PR #5431 threaded packetOffset and the correlation token through it and through Driver::draw so materialization/decode failures can preserve correlation identity. The materialization calls and try/catch wrappers are the instrumentation path, and they keep the original early-return behavior. If this second parameter pass is still judged too broad after review, I can split it into a refactor(agc) preparatory change, but it currently supports the request/materialization failure records described in the PR.

C. Silent allocation failures

The previous empty catch now emits draw-token-note-failed token=... through the best-effort logging path, so dropped correlation data is observable rather than indistinguishable from a complete trace. As before, this cannot throw past the call site, append to the reordered token list before success, or block QueueSubmit/draw success.

Validation

• Local Windows Release build: passed, including core/libs/libs/libSceAgcDriver.prx and NID patching.
• CaptureTrace focused tests: 8/8 passed.
• git diff --check: passed.
• Conventions check: passed.
• GitHub checks on the new head: conventions, eligibility, and report passed; platform jobs remained skipped, as in the earlier runs.

#5238 overlap

I compared the overlap with PR #5238. It remains complementary: #5238 exposes per-draw shader address/context details, while this PR adds token lifecycle and submission-batch correlation. If #5238 lands first, this PR should adapt to its address formatting rather than duplicating it. No changes were made to #5238's branch.

No claim of GPU/game-runtime validation is made.

[adriadam10]

Thanks for the quick follow-ups on the dedup and the empty catch. Blocking points:

• Conflicts with main in code, not just adjacency: Execution/src/Driver/Draw/Draw.cpp and Draw/DrawCapture.cpp (main's 73a808e changed how draw stages use their shared compiled result). Driver::materializeDrawStage has to be redone on top of that code, and the draw tests rerun after the rebase.
• Split the refactor out (as @CnRJay asked): moving per-stage materialization into Driver::materializeDrawStage (27 parameters, several out-parameters) rewrites the hottest function in the driver, and it is the part that conflicts with #5238, #5364, #5441, #5609 and #5656. Send that as its own refactor(agc) PR that is behaviour-neutral and has its own tests; then keep this PR to the token passed as one extra argument through the existing code.
• Evidence for the claim: "Unblocks PPSA02225 #4461" is not backed by anything in the PR. Show a trace from that title that reveals a failure that is not visible today, or drop the claim.

Welcome back once it is rebased and the refactor is separated.

[adriadam10]

Thanks for the rebase. Re-checked on the head merged onto current main; the conflict and the materializeDrawStage split points are resolved (main already has that function). Blocking points left:

• The PR's own tests fail (build + ctest of libSceAgcDriver on the merged tree): agc_driver_draw_shader_failure ("the capture trace received no draw-shader-request-failed record", log shows unsupported SOP1 opcode 2), agc_driver_draw_shader_failure_successful, and agc_driver_draw_shader_failure_request_correlation_disabled ("driver shutdown failed unexpectedly"). The agc_capture_trace_* tests pass. Please run them against a rebase on current main and fix whatever changed under them.
• Evidence for the claim: "Unblocks PPSA02225 #4461" is still not backed by anything in the PR. Show a trace from that title that reveals a failure that is not visible today, or drop the claim.

Welcome back once the tests pass on the rebased head.

[sktharunbalaji13-cmd]

Rebased cleanly on current \main\ (\80249ea54) and resolved all conflicts.

All 14 unit tests in \libSceAgcDriver\ now pass 100%:

• All 8 \�gc_capture_trace_*\ tests pass.
• All 6 \�gc_driver_draw_shader_failure_*\ subtests pass:
  1. \�gc_driver_draw_shader_failure\ (Passed)
  2. \�gc_driver_draw_shader_failure_successful\ (Passed)
  3. \�gc_driver_draw_shader_failure_correlation_disabled\ (Passed)
  4. \�gc_driver_draw_shader_failure_request_correlation_disabled\ (Passed)
  5. \�gc_driver_draw_shader_failure_materialization\ (Passed)
  6. \�gc_driver_draw_shader_failure_materialization_correlation_disabled\ (Passed)

Test fixture adjustments:

• Backed the color target in \DrawShaderFailure.cpp\ with a page-aligned buffer registered with \GuestAllocations::Mutation().Add\ and enabled render-target color mode (\

[adriadam10]

Re-checked the new head against our last comment: the tests point and the PPSA02225 claim are both resolved (claim dropped), and conventions pass. One blocking point left:

• DrawPipeline.cpp:78: if (stopped) return; in Enqueue is unrelated to tracing and hides a fatal failure. Once a commit fails fatally the worker sets stopped = true and stores failure; before this change the next Enqueue called rethrowFailure() and surfaced it to the submitting packet. With the early return, draws are dropped silently and failure is never rethrown. Please remove it. If a test needs a clean shutdown after a fatal failure, make the test handle that (or send it as a separate PR with a test that fails without it).
• Minor: Lifecycle.cpp:10 adds an include of DrawPipeline.hpp that Lifecycle.cpp:5 already has; drop it.

Welcome back once the early return is gone and the tests still pass.

[adriadam10]

The merge queue's Linux Build run on the current head (run 39157472408, PR merged onto main) fails four of this PR's own tests, deterministically:

• agc_driver_draw_shader_failure_successful, agc_driver_draw_shader_failure_correlation_disabled, agc_driver_draw_shader_failure_materialization, agc_driver_draw_shader_failure_materialization_correlation_disabled.
• The log shows [gpu] driver failure: AGC driver: guest memory has no write permission (the colour-target writability check at Graphics/src/Draw.cpp:2566) and AGC graphics: the submission failure lost the original shader-request error. The first test and the request_correlation_disabled one pass, so it looks tied to submitting further draws into the same ColorMemory after an earlier one in the process.

Please rebase on current main, run ctest -R agc_driver_draw_shader_failure (Linux/lavapipe if you can, or check the Windows run too) and make the tests independent of each other, e.g. a fresh writable colour buffer per case, or find why the buffer loses write permission. (agc_driver_depth_bias_per_face also fails in that run but is not caused by this PR.)

[sktharunbalaji13-cmd]

Rebased on current \main\ (\676d423968) and resolved the test failures:

• Root cause of \guest memory has no write permission\ on Linux: In \DrawPacket, register \