Skip to content

perf(shader): take the f32 mul, fma and add result fixups only when needed - #5400

Closed
ErenYeager1836 wants to merge 1 commit into
boykopovar:mainfrom
ErenYeager1836:carry/shader-f32-rare-fixups
Closed

ErenYeager1836 wants to merge 1 commit into
boykopovar:mainfrom
ErenYeager1836:carry/shader-f32-rare-fixups

Conversation

@ErenYeager1836

Copy link
Copy Markdown
Contributor

What

v_mul_f32, v_fma_f32, v_add_f32 and v_sub_f32 compute two fixups on every result: the AMD NaN result (the first NaN operand, quieted, or the default NaN) and, for products, the flush of results that are tiny after rounding to 24 bits. Both change the result only when it is a NaN or its magnitude is at most 2^-126. They were straight-line selects, so a v_mul_f32 took 16.5 SASS instructions on an RTX 4060 where the multiply is one.

The translator now wraps the fixed result in a select that takes it only when the host result is NaN (add and sub) or NaN or at most 2^-126 in magnitude (mul and fma, round to nearest even). That select is flagged as rarely true, and the SPIR-V backend moves the instructions used only by its rare arm into a branch. The rare arm is the same IR as before, so results are unchanged. Loop headers, continue targets and values shared between the two halves of a wave64 program keep plain selects.

Carried from @karanvora2599 with original attribution preserved.

Tested

  • Windows 11: python tools/check_conventions.py --base upstream/main passes with 0 errors.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: none
  • AI-assisted: yes

…eeded

v_mul_f32, v_fma_f32, v_add_f32 and v_sub_f32 compute two fixups on
every result: the AMD NaN result (the first NaN operand, quieted, or the
default NaN) and, for products, the flush of results that are tiny after
rounding to 24 bits. Both change the result only when it is a NaN or its
magnitude is at most 2^-126. They were straight-line selects, so a
v_mul_f32 took 16.5 SASS instructions on an RTX 4060 where the multiply
is one.

The translator now wraps the fixed result in a select that takes it only
when the host result is NaN (add and sub) or NaN or at most 2^-126 in
magnitude (mul and fma, round to nearest even). That select is flagged
as rarely true, and the SPIR-V backend moves the instructions used only
by its rare arm into a branch. The rare arm is the same IR as before, so
results are unchanged. Loop headers, continue targets and values shared
between the two halves of a wave64 program keep plain selects.

A dependent chain of 2048 v_mul_f32 runs 2.2 times as fast, v_fma_f32
1.85 times and v_add_f32 1.13 times; on all-NaN inputs v_mul_f32 is 6%
slower. Static SASS grows, since both paths are compiled.
@github-actions github-actions Bot added the shader core/shader label Oct 13, 2026
@karanvora2599

Copy link
Copy Markdown
Contributor

@ErenYeager1836, this is #4322's commit without #4287 under it. I closed #4322 because branching the rare arms makes pipeline creation much slower (details in #4322):

  • simonbbby measured the large f8d1fab9 pixel shader of Surround Sound Studio (PPSA99051). Its SPIR-V grew 34%, and the RTX 3090's pipeline build went from 2.2 s to 3.8 s. RADV cold runs went from 3 of 3 passing to 0 of 4 at the title's normal wait.
  • On an RTX 4060, pipeline creation for independent f32 chains went up 47% at 1,024 ops and 119% at 4,096 ops.

NVIDIA's pipeline creation time is already the worst stall we have (#2292), so please close this one.

RINNECODER pushed a commit to RINNECODER/AnyPS5 that referenced this pull request Oct 13, 2026
…eeded

v_mul_f32, v_fma_f32, v_add_f32 and v_sub_f32 compute two fixups on
every result: the AMD NaN result (the first NaN operand, quieted, or the
default NaN) and, for products, the flush of results that are tiny after
rounding to 24 bits. Both change the result only when it is a NaN or its
magnitude is at most 2^-126. They were straight-line selects, so a
v_mul_f32 took 16.5 SASS instructions on an RTX 4060 where the multiply
is one.

The translator now wraps the fixed result in a select that takes it only
when the host result is NaN (add and sub) or NaN or at most 2^-126 in
magnitude (mul and fma, round to nearest even). That select is flagged
as rarely true, and the SPIR-V backend moves the instructions used only
by its rare arm into a branch. The rare arm is the same IR as before, so
results are unchanged. Loop headers, continue targets and values shared
between the two halves of a wave64 program keep plain selects.

A dependent chain of 2048 v_mul_f32 runs 2.2 times as fast, v_fma_f32
1.85 times and v_add_f32 1.13 times; on all-NaN inputs v_mul_f32 is 6%
slower. Static SASS grows, since both paths are compiled.

Ported from upstream PR boykopovar#5400.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj
RINNECODER added a commit to RINNECODER/AnyPS5 that referenced this pull request Oct 13, 2026
* perf(shader): pick the f32 NaN result with one quiet and one fallback select

nanResultF32 gives the f32 add, sub, subrev, mul and fma forms, the legacy
mul, mad and fma forms, v_div_fmas_f32 and v_fma_mix_f32 with f32 sources
the AMD NaN result: the first NaN source, quieted in IEEE mode, else the
default NaN 0xffc00000 when the result is NaN, else the result. It started
from the fallback and then, for each source, tested it for NaN with an
integer and and compare, quieted it with its own or, and selected it. The
addend's test was also combined with the inverse of the invalid-product
flag.

The chain now carries only source bits. The addend starts it, replaced by 0
when the product is 0 * inf. Each earlier source that FPIsNan32 finds NaN
takes its place. One quiet and one select then choose between that source,
when it is NaN, and the fallback. The candidate is NaN exactly when some
eligible source is, and it is then the first one, so every result is the
same.

agc_driver_f32_arithmetic_nan already checks the source order and the 0 *
inf product with a NaN addend for every form.

Ported from upstream PR boykopovar#4287.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* perf(shader): drop the output flush of f32 products

In a FLOAT_MODE that flushes f32 denormals, writeOperand flushes every
f32 result, and DenormalFlushEliminator removes the flush only when the
value can't be a denormal. It had no rule for products, so every
v_mul_f32 kept its output flush, although the tiny-product flush before
it already leaves no denormal.

The eliminator now proves a product denormal-free from three facts:

- The tiny-product flush of a multiply (round to nearest even, no
  addend) returns a signed zero or a product at least 2^-126 in
  magnitude, so its select is flagged notDenormal. An fma isn't
  flagged, since its addend can cancel the result into a denormal.
- The NaN select picks its first arm only when that arm is a NaN, so it
  is as denormal-free as its fallback.
- A product by a constant that is ±0 or at least 1 in magnitude, of a
  denormal-free factor, isn't a denormal.

A dependent chain of 2048 v_mul_f32 on an RTX 4060 runs 9% faster.
v_mul_legacy_f32 loses the same flush. Results are unchanged.

The f32 denormal flush test gets rows where a nonzero addend cancels a
normal product into an exact denormal, which the fma output flush must
turn into a signed zero. No row covered that before.

Ported from upstream PR boykopovar#4539.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* perf(shader): treat f32 min, max, med3, floor and ceil results as denormal-free

In the float modes that flush f32 denormals, writeOperand flushes the result
of v_min/max/min3/max3/med3_f32 and v_floor/ceil_f32. DenormalFlushEliminator
drops a flush only when it can show the value isn't a denormal, and it had
no rule for these opcodes, so their result flush stayed even when every
operand had already been flushed.

FPMin32, FPMax32, FPMinTri32, FPMaxTri32 and FPMedTri32 return one of their
operands' bits, or the and or or of two zeros, so the result is denormal-free
when every operand is. FPFloor32 and FPCeil32 return an integer, a zero, an
infinity or a NaN whatever their operand. In IEEE mode the min and max
results and the floor and ceil results go through selects with a quieted
operand, which the eliminator can't follow, so their flush stays there.

agc_denormal_flush_elimination covers the five min and max forms with
flushed and with unknown operands, floor and ceil of an unknown value, and a
max carried around a loop.

Ported from upstream PR boykopovar#4490.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* perf(shader): treat f32 fract, sin and cos results as denormal-free

In the float modes that flush f32 denormals, writeOperand flushes the result
of v_fract_f32, v_sin_f32 and v_cos_f32 after flushing their input.
DenormalFlushEliminator had no rule for FPFract32, FPSin or FPCos, so the
result flush stayed.

None of the three returns a denormal for an input that isn't one.
EmitFPFract32 takes fract(|x|), which is |x| below 1 and otherwise a multiple
of the input's ulp, and 1 - fract, which is at least 2^-24. EmitFPSin and
EmitFPCos reduce the input exactly with x - roundEven(x), so the cycle and
its distance from a zero of the function are 0 or at least 2^-126. Near a
zero, TrigLinearNearZero returns the distance times about 6.28. Elsewhere
the result is within 4 ulps of a series value of at least about 1.5e-3. In
IEEE mode floatUnary quiets a NaN input with an or, which the eliminator
can't follow, so the flush stays there.

agc_denormal_flush_elimination covers the three opcodes with a flushed and
with an unknown input.

Ported from upstream PR boykopovar#4737.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* perf(shader): drop flushes of f32 results that cannot be denormal

DenormalFlushEliminator removes f32 denormal flushes on values it can prove are not denormal. Recognises FPTrunc32, FPRoundEven32, square roots (FPSqrt), reciprocal square roots (FPRecipSqrt32), and bitwise operations that set exponent bits (BitwiseOr32) or mask to signed zeros (BitwiseAnd32). Adds unit tests in DenormalFlushElimination.cpp.

Ported from upstream PR boykopovar#5086.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* fix(shader): take the fused result when an f16 fma's sum is exact

fmaF16RoundedToOdd adds the f32 product and the addend separately and uses
TwoSum to round to odd when the sum is inexact. When the error is zero it
returned the separate sum. On MoltenVK (Apple M3 Pro) the separate multiply and
add produce +0 where the IEEE result is -0: agc_driver_dpp16_operands got 0x0000
instead of 0x8000 in the high half of lane 1's row_shr v_pk_fmac_f16.

The exact case now returns an f32 fma of the same operands. The product of two
f16 values is exact in f32 (11-bit significands, exponents well inside the f32
range), so when TwoSum's error is zero the separate sum and the fused result are
the same IEEE value, including the sign of zero; the change only matters on
devices whose separate multiply and add do not keep it. The inexact path is
unchanged.

Ported from upstream PR boykopovar#4293.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* perf(shader): take the f32 mul, fma and add result fixups only when needed

v_mul_f32, v_fma_f32, v_add_f32 and v_sub_f32 compute two fixups on
every result: the AMD NaN result (the first NaN operand, quieted, or the
default NaN) and, for products, the flush of results that are tiny after
rounding to 24 bits. Both change the result only when it is a NaN or its
magnitude is at most 2^-126. They were straight-line selects, so a
v_mul_f32 took 16.5 SASS instructions on an RTX 4060 where the multiply
is one.

The translator now wraps the fixed result in a select that takes it only
when the host result is NaN (add and sub) or NaN or at most 2^-126 in
magnitude (mul and fma, round to nearest even). That select is flagged
as rarely true, and the SPIR-V backend moves the instructions used only
by its rare arm into a branch. The rare arm is the same IR as before, so
results are unchanged. Loop headers, continue targets and values shared
between the two halves of a wave64 program keep plain selects.

A dependent chain of 2048 v_mul_f32 runs 2.2 times as fast, v_fma_f32
1.85 times and v_add_f32 1.13 times; on all-NaN inputs v_mul_f32 is 6%
slower. Static SASS grows, since both paths are compiled.

Ported from upstream PR boykopovar#5400.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* fix(shader): read lane 63 of half-wave reductions whose keys exclude helper lanes

On a 32-lane host, a wave64 pixel program's v_readlane_b32 of lane 63 is emitted only when its half-wave scan's keys are proven zero for lanes that did not enter. The proof walk accepted the not-helper test (IEqual32 of HelperInvocation and 0) only inside ballots, not as a lane bit, and it searched conjunctions depth first: a key mask like exec & ~vcc inside a loop spent the 512-step budget on the loop's exec phis before it reached the helper test. Accept the helper test as a lane bit, and look for a false or helper conjunct before walking an operand. Resident Evil Village's wave-wide OR reductions (DPP row shifts, v_permlanex16, v_readlane of lanes 31 and 63) failed this way.

Ported from upstream PR boykopovar#5189.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* test(shader): cover the search order of the half-wave key proof

The proof that a half-wave scan's keys are zero for lanes that did not
enter walks a conjunction depth first under a 512-step budget. Narrowing
exec with 600 compares before the scan puts the helper-lane test at the
bottom of a 600-deep LogicalAnd chain in the keys: walking the first
operand first spends the whole budget before reaching it, and the
v_readlane_b32 of lane 63 is refused. Looking for a false or helper
conjunct before walking an operand finds it, so lane 31 becomes a
subgroup OR on a 32-lane host. Resident Evil Village's wave64 pixel
programs key these scans by a loop's exec ahead of the helper test.

Ported from upstream PR boykopovar#5189.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* perf(shader): find the ds_permute_b32 source lane with ballots

ds_permute_b32 sends each active lane's data to lane
((addr + offset) >> 2) & 31 of its 32-lane group; the highest sending
lane wins and a lane nobody writes to gets 0. It was emitted as one
shuffle of the data, the address and the EXEC bit from each of the 32
source lanes: 96 shuffles and 144.5 SASS instructions per instruction
on an RTX 4060 when the address changes between instructions.

Each receiving lane now finds its writers directly: the ballot of the
active lanes, intersected with the ballot of the lanes whose target has
bit b set, or its complement, for each of the 5 target bits. The
highest set bit is the winning lane, and one shuffle fetches its data.
Results are unchanged. A chain of 256 dependent ds_permute_b32 runs
4.7 times faster.

New test agc_driver_ds_permute_lanes compares wave32 and wave64
programs with random targets, heavy collisions and run-time EXEC masks
against a model of the rule.

Ported from upstream PR boykopovar#5334.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* perf(shader): mark runtime metadata buffers read-only

Ported from upstream PR boykopovar#5297.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* perf(shader): fence a CAS-loop LDS atomic once

SharedAtomicUpdate emitted a workgroup acquire-release fence after AtomicUpdate,
which already ends its compare-and-swap loop with the same fence for LDS, so
ds_rsub, ds_add_f32, ds_cmpst, ds_mskor, ds_wrap and the LDS float min/max
fenced twice. The second fence is now emitted only for GDS, where the loop's
own fence is device-scoped on uniform memory and differs from it.

Ported from upstream PR boykopovar#5399.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

* chore(shader): bump the shader disk cache format for the float peephole ports

The ported float, denormal and wave-lane changes alter the emitted shader
code, so cached entries from before them must not be reused.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgW4n3qYgi5jAAohEmXRQj

---------

Co-authored-by: Karan Vora <52519664+karanvora2599@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Friesmaniac <181011464+Friesmaniac@users.noreply.github.com>
Co-authored-by: Utku Halis <personal@utkuhalis.com.tr>
Co-authored-by: Yunus Shaikh <75296019+EagleBearerYS@users.noreply.github.com>
Co-authored-by: Sanxchep Sharma <sanxchep@gmail.com>

@CnRJay CnRJay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding to the comment above. This trade-off is measured, and it went the wrong way. Moving the rare arm into a branch removes a few ALU instructions per v_mul_f32 at run time. It also turns every one of these ops into a selection construct: thousands of extra basic blocks in large shaders, which is what drives the 34% SPIR-V growth and the 47–119% longer pipeline builds reported in #4322. Driver compilers also tend to flatten short branches like these back into selects (if-conversion), so the run-time gain may not survive anyway, while the compile-time cost is always paid.

The description's "Tested" covers only the conventions check, with no GPU and no shader. Since this re-opens a change closed for a measured compile-time regression, it would need new measurements that answer #4322: pipeline creation time on the same large shaders and GPU time of a title's frame, on at least NVIDIA and RADV, before it could be considered.

The same applies to #5334, #5337 and #5399, which also carry closed PRs over unchanged. If there's new evidence for any of them, putting it in the PR description up front would save the reviewers a round trip.

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Closing: this is #4322's commit, and #4322 was closed because branching the rare arms makes pipeline creation much slower (measurements there, as karanvora2599 says above). Reopen with numbers from a title scene if that changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

shader core/shader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants