Skip to content

feat(shader): implement triangle-patch tessellation - #1491

Merged
adriadam10 merged 4 commits into
boykopovar:mainfrom
Celegans12:feat/shader-tessellation-triangle-patches
Oct 11, 2026
Merged

adriadam10 merged 4 commits into
boykopovar:mainfrom
Celegans12:feat/shader-tessellation-triangle-patches

Conversation

@Celegans12

@Celegans12 Celegans12 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What

Every tessellated draw throws today: DecodeShaderStages rejects PRIMGEN_PASSTHRU_EN, the local program ends in an unresolvable s_setpc_b64, and hull and domain programs keep unlowered ring bases. Since #639 a title's local programs also stop registration.

  • The tessellation path accepts LS, HS and an NGG passthrough ES, reads PARTITIONING and TOPOLOGY as 3-bit fields, throws on the other VGT_TF_PARAM bits except DISTRIBUTION_MODE, and needs HOS levels that keep [1, 63] (AMD's Sea Islands 3D register reference); 0x286-0x287 join the draw key.
  • Local and domain (ES_EN = DS) programs are prepared with their stage group, not at registration. ResolveGraphicsStagesAbi merges the headers over one set of defaults, which used to reset the hull's LS_HS_CONFIG and TF_PARAM. A local program the hull header leaves at address 0 is prepared at its first draw.
  • The local program ends at its jump into the hull half. TessellationLowering lowers LDS and ring addresses affine in the control point and relative index, with strides from the code; accesses outside the patch, per-patch writes not from control point 0, other relative-index uses and a non-passthrough primitive export throw.
  • Hull lanes read their own saved exec bit, also from fix(shader): mask EXEC copies to the host subgroup's lanes #1264's masked copies; other masks throw; factors clamp to 63.
  • Quads, isolines, CCW output and domain patch-constant reads still throw. TechnicalDebt (Functional): unwritten rings, NaN inner factors, DISTRIBUTION_MODE, the local-to-hull handoff, late local programs.
  • PPSA26344 v1.008: 1,520 G-buffer draws per run (0x200210d, 0xc355, 0x40049); its LS headers have no 0x148 and its HS headers set 0x148 to 0.
  • feat(agc): accept graphics programs without an AGC shader header #1732 and perf(agc): dispatch-cache recipe hits as ordered commit items #1747 rewrite the program lookup in decodeDraw.

Tested

  • Windows 11, MinGW-w64 GCC 15.2.0, based on main 4b6ab6b: configure -DBUILD_TESTING=ON, full build, --target libs, ctest:
  • 480/480 tests pass (ctest --output-on-failure)
  • Same build on lavapipe (Mesa 26.2.4 llvmpipe, 8-wide subgroups, VK_DRIVER_FILES): 479/479 pass; agc_driver_flat_store is left out there because it takes several minutes on Windows llvmpipe
  • New agc_driver_tessellation: level-1, 0 and 5 patches through hand-assembled programs (coverage, culling, interior points, winding), six refusals, and title-shaped LS/HS/DS headers through registration and ResolveGraphicsStagesAbi; fails on main.
  • agc_driver_prepared_graphics: a hull ABI without its local program, then that program prepared at its first draw; new rows in agc_driver_graphics; both fail on main.
  • PPSA26344 v1.008 before the perf(agc): baseline shader recompilation on registration #639 port: no routing skip left (1,600 before). Offline, its LS/HS/DS headers now register and the prim-state ABI stops at the domain program's T# record refusal (pc 0x19c).
  • Not tested: Linux (CI), console behaviour.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • AI-assisted: yes

@github-actions github-actions Bot added build shader core/shader docs agc libSceAgc* labels Oct 8, 2026
@Celegans12
Celegans12 force-pushed the feat/shader-tessellation-triangle-patches branch 2 times, most recently from 9849333 to 680c202 Compare October 8, 2026 14:20
@yahya308

yahya308 commented Oct 8, 2026

Copy link
Copy Markdown

Tested on top of main 9e2ffe0d, Windows 11, MinGW-w64 GCC 16.2.0, full ctest with ANYPS5_REQUIRE_VULKAN=1 --repeat until-fail:3, NVIDIA RTX 4070 Laptop (32.0.16.1074) and AMD Radeon 780M (32.0.31041.1004). vopc_compare_table fails on the 780M on main too (#825, #1473).

Head 680c2028: NVIDIA 458/458, 780M 457/458. The new agc_driver_tessellation ran and passed on both.

AI-assisted: yes (Claude Code).

@github-actions github-actions Bot added needs rebase Conflicts with main, rebase needed labels Oct 8, 2026
@Celegans12
Celegans12 force-pushed the feat/shader-tessellation-triangle-patches branch from 680c202 to 7d8b561 Compare October 8, 2026 20:13
@github-actions github-actions Bot added needs rebase Conflicts with main, rebase needed and removed needs rebase Conflicts with main, rebase needed labels Oct 8, 2026
@Celegans12
Celegans12 force-pushed the feat/shader-tessellation-triangle-patches branch from 7d8b561 to 55473d0 Compare October 9, 2026 03:20
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 9, 2026

@adriadam10 adriadam10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agc_driver_tessellation crashes on RADV, so this can't go in as is. The crash happens inside the driver, but on main these draws throw a clean error, while with this PR the whole process dies on the default Linux AMD driver.

  • Where: Linux, Mesa 26.2.4 RADV, Radeon 660M (gfx1035, RDNA2, 64-wide subgroups). The PR is merged onto main 787239e. The test segfaults in vkCreateGraphicsPipelines on its first draw (RequirePatches → Graphics::Pipeline::Pipeline). It fails the same way under ACO and under RADV_DEBUG=llvm, nongg and nooptimizations. It passes on CI's lavapipe (Mesa 25.2.8).
  • The input is valid: spirv-val --target-env vulkan1.3 accepts all four modules, and the Khronos validation layer (1.4.357) reports nothing before the crash. So this is a RADV bug, but this PR is the first thing that sends tessellation pipelines to it.
  • What triggers it (I swapped single modules with MESA_SPIRV_READ_PATH): the crash needs both the hull module and the domain module this PR emits. If either one is swapped for a hand-written GLSL stage with the same interface, the crash goes away. If both are run through spirv-opt -O (which folds the runtime-computed indices, e.g. (32752 - 32752) >> 4 for the patch output, and the UMod/UDiv component indices into constant access chains), the crash also goes away, but then the level-0 patch is not culled. I haven't checked whether that last part comes from spirv-opt or RADV.
  • Suggested direction: reduce it to the construct RADV can't handle. The first suspects are the dynamically indexed components of the per-vertex and patch outputs/inputs when the address is already a constant. Emitting constant access chains in that case would avoid it, and would also give simpler SPIR-V. Then check that the level-0 patch is still culled on RADV. If you have no AMD GPU on Linux, I can rerun a new head here. A reduced case is also worth filing with Mesa.

To reproduce: MESA_SPIRV_DUMP_PATH=<dir> ctest -R agc_driver_tessellation on RADV dumps the four modules (.vert/.tesc/.tese/.frag). Put modified copies with the same names in a directory and point MESA_SPIRV_READ_PATH at it to swap them.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 10, 2026
tibzejoker added a commit to tibzejoker/AnyPS5 that referenced this pull request Oct 10, 2026
…eam)

- AGC shader binary type 3 (Hs, front+back bundle) is accepted where HsBack (7) is
- a type 3 header over a registered HsBack keeps the HsBack snapshot (its user data)
  and only contributes its registers to the graphics ABI
- port of PR boykopovar#1491's front-half decode: the Local stage's s_setpc_b64 s[6:7]
  hand-off to the back half ends the program

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1W4jdNSKTV7xwuS6xQPJV
tibzejoker pushed a commit to tibzejoker/AnyPS5 that referenced this pull request Oct 10, 2026
…ar#1491 onto main)

Port of boykopovar#1491 (Celegans12) onto main 197ffac. Conflicts
resolved: DrawKeyRegisters keeps main's 0x005 range and adds 0x286-0x287;
the Local front-half decode moved into the cached frontend; the Local
prologue keeps main's drawIndex with the RelativeIndex tessellation base.

Known failure: agc_driver_tessellation ("push constant range crosses a
stage slot") since main's fixed push slots (boykopovar#2661).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1W4jdNSKTV7xwuS6xQPJV
tibzejoker added a commit to tibzejoker/AnyPS5 that referenced this pull request Oct 10, 2026
…eam)

- AGC shader binary type 3 (Hs, front+back bundle) is accepted where HsBack (7) is
- a type 3 header over a registered HsBack keeps the HsBack snapshot (its user data)
  and only contributes its registers to the graphics ABI
- port of PR boykopovar#1491's front-half decode: the Local stage's s_setpc_b64 s[6:7]
  hand-off to the back half ends the program

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1W4jdNSKTV7xwuS6xQPJV
tibzejoker added a commit to tibzejoker/AnyPS5 that referenced this pull request Oct 10, 2026
…eam)

- AGC shader binary type 3 (Hs, front+back bundle) is accepted where HsBack (7) is
- a type 3 header over a registered HsBack keeps the HsBack snapshot (its user data)
  and only contributes its registers to the graphics ABI
- port of PR boykopovar#1491's front-half decode: the Local stage's s_setpc_b64 s[6:7]
  hand-off to the back half ends the program

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1W4jdNSKTV7xwuS6xQPJV
Celegans12 and others added 4 commits October 11, 2026 11:25
DecodeShaderStages accepts LS, HS and an NGG passthrough ES, reads the
3-bit VGT_TF_PARAM fields, refuses its other bits but DISTRIBUTION_MODE,
and needs HOS levels that keep [1, 63]; 0x286-0x287 join the draw key.
Local and domain programs are prepared with their stage group: its
headers are merged over one set of defaults, and a local program left
unbound by the hull header is prepared at its first draw.

The local program ends at its jump into the hull half.
TessellationLowering lowers LDS and ring addresses that are affine in
the control point and relative index to tessellation interfaces with
strides from the code, and refuses accesses outside the patch, per-patch
writes from other control points and a modified primitive export. Hull
lanes read their own saved exec bit, also of host-subgroup-masked
copies; other masks are refused, gl_InvocationID loads as a scalar and
factors clamp to 63.
…reparation at use

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ation

Co-Authored-By: Claude <noreply@anthropic.com>
@adriadam10
adriadam10 force-pushed the feat/shader-tessellation-triangle-patches branch from 55473d0 to c8f8014 Compare October 11, 2026 09:40

@adriadam10 adriadam10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there was no new push, I went through the points from my last review myself and pushed them to this branch on top of a rebase onto current main:

  • Rebase: resolved the conflicts with #639's shared frontend (DecodeStageProgram now feeds PreparedControlFlow), the embedded-fetch drawIndex (v3 keeps the relative index), the per-address shader registry, and main's preparation at first use, which replaces the special case for local programs. DrawKeyRegisters is now 48.
  • Tests: Tessellation.cpp uses the fixed per-stage push constant slots (PixelPushLayout/PixelPushOffset). PreparedGraphics.cpp now expects a second tessellation configuration of a local program to be prepared at its draw rather than throw, which matches what main does now.
  • RADV crash: reduced with MESA_SPIRV_READ_PATH. RADV 26.2.4 crashes when the slot index of a hull per-vertex input ((address % stride) >> 4) is dynamic. A dynamic component index is fine. The lowering always produces invocation * stride + c for those reads, so TessellationPointer now indexes them as [invocation + c / stride][c % stride / 16][c % 16 / 4] with constant slot and component. agc_driver_tessellation passes on RADV (660M, gfx1035), including the level-0 patch being culled. It also passes on CI's lavapipe (Mesa 25.2.8), and the Khronos validation layer reports nothing.
  • TechnicalDebt: local programs are now prepared at their first draw once per tessellation configuration, so a later configuration no longer throws.

Checked: scoped build + ctest on RADV, 570/570 passed (VideoOut window tests skipped, no display).

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 11, 2026
@adriadam10
adriadam10 merged commit b70a0e2 into boykopovar:main Oct 11, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* build docs shader core/shader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants