fix(shader): round normalized storage image writes from the exact product

[tibzejoker]

What

Vulkan lets a device return either neighbour when it converts a float to a normalized texel ("implementations should round to nearest"), and devices differ in practice. For a stored value whose product with 255 is 123.50003, an RTX 4090 and an Intel UHD 770 both store 123 instead of 124. On GTA V's FSR1 RCAS compute pass (PPSA04264, compute 0x25b393000, RGBA8 output), the two vendors stored 112 different bytes for the same inputs.

Writes to unorm and snorm storage images whose components share one width (8_UNORM, 16_16_UNORM, 8_8_8_8_UNORM/SNORM, 16_16_16_16_UNORM, ...) now:

• clamp the value to [0, 1] (or [-1, 1]), with NaN mapped to 0;
• round value * scale in the shader to the level nearest the EXACT product, ties to even. fl32(value * scale) can land exactly on a .5 when the exact product does not; the fma residual fma(value, scale, -p) breaks that tie by its sign. RDNA 2 (RX 6700 XT, native image_store, measured by @thuanquang) and Apple M2 (@jotitadev) both round this way. The residual is decorated NoContraction: without it the Intel UHD 770 does not evaluate the fma exactly;
• store level * c, where c is the smallest float not below 1 / scale.

The exact product of that stored float with the scale lies in [level, level + 0.01) for 8-, 10- and 16-bit unorm and snorm. I checked every level with exact rational arithmetic. A device that rounds to nearest and a device that truncates therefore both store level.

The format comes from the image descriptor through specialization constants (StoreQuantizeBase, four per written storage image), so float formats, packed formats and format codes outside the format table keep the plain write. Registration modules, tables and atomics are unchanged.

Tested

Windows 11, MinGW GCC 15.2:

• New test agc_driver_image_store_unorm_rounding. It stores values next to the midpoints between levels, edge values and NaN/infinities to 8_UNORM, 16_16_UNORM, 8_8_8_8_UNORM, 8_8_8_8_SNORM and 16_16_16_16_UNORM images.
  • On main it fails on both GPUs: image_store of 8_UNORM: texel (6, 0) component 0 is 0x7b, expected 0x7c.
  • With this change it passes on both (3 runs each, ANYPS5_NO_SHADER_CACHE=1). The expected level is nearbyint of the product computed in double, exact for these inputs.
  • The first version of this PR (nearest even of fl32(value * scale)) fails it: texel (10, 0) component 0 is 0xce, expected 0xcd (exact product 205.49999..., RDNA 2 stores 205).
• ctest -R ^agc_ on the RTX 4090: 727/728. The failure is agc_driver_bda_device, which fails on main too.
• GTA V compute shaders, replayed offline from dumped requests and compared byte for byte between the two GPUs: RCAS now stores identical bytes on NVIDIA and Intel.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: NVIDIA GeForce RTX 4090 (616.92, Windows 11); Intel UHD Graphics 770 (32.0.101.6129, Windows 11); AMD Radeon 860M (32.0.13058.8001, Windows 11, by @LeandroLP23, rerun on 61340e2: passes, the first version's shader fails); rounding rule checked on RDNA 2 hardware (RX 6700 XT, HIP) by @thuanquang and on Apple M2 by @jotitadev; Apple M2 Max (MoltenVK 1.4.2 and main, fast math 0/1/2) rerun on 61340e2 by @jotitadev: passes, the first version's shader fails; AMD Radeon RX 7900 XT (RADV NAVI31, Mesa 25.2.8, Linux) by @DavidAmsellem: full suite, no new failure
• AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

https://claude.ai/code/session_01XBXM4pgkMHYEtqN1EQni2P

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 907	core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 5021	core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 5371	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/PipelineSpecialization.hpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 5500	core/shader/recompiler/PipelineSpecialization.hpp
PR 5609	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp
PR 6280	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/PipelineSpecialization.hpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 6412	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp
PR 6488	core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 6539	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp
PR 6947	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp
PR 6959	core/shader/recompiler/PipelineSpecialization.hpp
PR 7085	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/PipelineSpecialization.hpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7223	core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7332	core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7395	core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7642	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/PipelineSpecialization.hpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7689	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp

[tibzejoker]

The overlap with #7085 is textual only: both add a constant next to MipCountBase in PipelineSpecialization.hpp. The ranges do not collide (LevelsPastViewBase = 8384 there, StoreQuantizeBase = 12288 + 4 per image here), so whichever merges second only needs the two lines kept side by side.

AI-assisted (Claude Code).

[tibzejoker]

A question for anyone with an RDNA hardware oracle (as used on #7671). This PR picks round-to-nearest-even of fl32(x * 255) for float-to-UNORM8 storage writes, because Vulkan leaves the choice to the device and NVIDIA and Intel disagree. I have not measured what RDNA itself stores.

An image_store of the values from agc_driver_image_store_unorm_rounding (the near-midpoint ones especially) to an 8_8_8_8_UNORM and a 16_16_16_16_UNORM image on native hardware would show whether the console rule is the same. If it differs, I'll change the rounding here to match it.

AI-assisted (Claude Code).

[LeandroLP23]

AMD run, since the description covers an RTX 4090. Radeon 860M (RDNA 3.5, driver 32.0.13058.8001), Windows 11 26300, WinLibs GCC 15.2.0 r7, Release, BUILD_TESTING=ON. Build: main 8a456334f with #7677, #7673, #7671, #7666, #7676, #7675, #7672, #7668, #7664 and #7658 merged together (each merge clean, built once).

agc_driver_image_store_unorm_rounding and agc_driver_bda_device pass 3 runs of 3 on AMD's proprietary driver.

AI-assisted

[jotitadev]

Apple GPU / MoltenVK run, with a data point for the vendor question. Without this PR, Apple stores the level nearest to the exact product x * scale, not to fl32(x * scale). That is a third behaviour next to NVIDIA and Intel. With the PR, agc_driver_image_store_unorm_rounding passes, and the full ^agc suite is unchanged.

Machine: Apple M2 Max, macOS 27.2, x86-64 build under Rosetta. Head 3e54993, PR diff applied cleanly. Drivers: MoltenVK 1.4.2 (Vulkan SDK 1.4.363.0) and MoltenVK main + SPIRV-Cross #2704.

Tested: the tree is main a373e85 + mugurc's macOS series (ci/macos-series-2) + the PRs needed to build on macOS (#6667, #6668, #7524, #6969, and two files of #7332), not plain main (it doesn't build on macOS yet). Also in the same build: #6606, #7683, #7696, #7692 and #7415 (they applied cleanly together), plus a test-only patch that lets nine AGC fixtures load the Vulkan loader on macOS (branch jotitadev:fix/agc-tests-vulkan-loader-macos). Base on that tree: 40 ^agc failures with MoltenVK 1.4.2 and 26 with MoltenVK main + #2704.

• With the PR: agc_driver_image_store_unorm_rounding passes 3 of 3 on both drivers. It still passes 3 of 3 with MVK_CONFIG_FAST_MATH_ENABLED=1, which makes MoltenVK compile with Metal fast math.

• With this PR's DescriptorBindingBuilder.cpp, PipelineSpecialization.hpp and SpirvImageEmitter.cpp reverted, the test fails on both drivers; the first mismatch is 8_UNORM texel (10, 0), 0xcd where 0xce is expected. I changed the check locally to log every mismatch instead of stopping at the first. 59 of the 736 written components differ, and the results are identical on both drivers:

  format	mismatches	example input	exact x * scale	Apple stored	test expects
  8_UNORM	4	0x1.9c9c9cp-1	205.4999954	205	206
  16_16_UNORM	7	0x1.159116p-5	2220.5000529	2221	2220
  8_8_8_8_UNORM	17	0x1.010102p-9	0.5000000296	1	0
  8_8_8_8_SNORM	16	-0x1.7cf9f4p-1	-94.5000004	-95	-94
  16_16_16_16_UNORM	15	0x1.cec1cep-7	925.4999769	925	926

  In each of these inputs, fl32(x * scale) is exactly .5, so round-to-nearest-even of the float product and rounding of the exact product give different levels. I recomputed the test's inputs offline: exactly 59 components fall in that case (4/7/17/16/15 per format), and Apple stored the level nearest to the exact product in all 59. The NaN, infinity, clamp and edge values match the test's expectations without the PR.

• Since both drivers give the same result, the conversion happens in Metal or the hardware, not in SPIRV-Cross. Apple seems to convert float to unorm/snorm with correct rounding. NVIDIA and Intel, as the description says, store 123 for 123.50003.

• On the RDNA question: if the console also rounds the exact product, the inputs above are where this PR's rule (RTNE of fl32(x * scale)) and the console would differ. The test already exercises them, so a hardware oracle run of it would settle the question.

• Full ^agc suite with the stack: the base failures on each driver, plus agc_driver_pooled_buffer_range and agc_performance_overlay, which only run because of the loader patch. Nothing from this PR.

GPUs tested: Apple M2 Max (MoltenVK 1.4.2; MoltenVK main + SPIRV-Cross #2704)

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

[tibzejoker]

Thanks @jotitadev, this is a real gap in the PR. I picked "round to nearest even of fl32(x * scale)" because NVIDIA and Intel both start from the f32 product. Apple rounds the exact product, and that is the more defensible rule: it does not depend on how the product was computed, and it is what "convert to the nearest level" means without a tolerance.

Rounding the exact product costs little in the shader. scale (2^n - 1, n <= 16) is exact in f32, and e = fma(x, scale, -p) gives the exact error of p = fl32(x * scale). When p - floor(p) is exactly 0.5, the sign of e picks the level (e > 0 -> up, e < 0 -> down, e == 0 -> even). Otherwise roundEven(p) is already the level nearest to the exact product. That is one fma and one compare per component, and the level * c store stays as it is.

What I don't know is what RDNA does, and the console is the reference. @simonbbby, could your RDNA 2 oracle run the stores of agc_driver_image_store_unorm_rounding, or just the 59 inputs above (image_store to an R8/RGBA8/RG16/RGBA16 UNORM and RGBA8 SNORM texture)?

• If RDNA rounds the exact product (like Apple), I'll switch the PR to the fma rule above.
• If it rounds fl32(x * scale), the PR already matches it, and Apple gets the emulation too.

Until there is data, I'll leave the PR as it is, so that it doesn't change twice.

Written with AI assistance (Claude Code).

[thuanquang]

@tibzejoker RDNA 2 answer: the hardware rounds the exact product, like Apple, not fl32(x * scale).

RX 6700 XT (gfx1031), native image_store through HIP (the shader runs as written, no Vulkan driver in between). For each format I generated the 128 values of agc_driver_image_store_unorm_rounding (Value(format, tid, index) for tid 0-31 and index 0-3, the edges, NaN, ±infinity and the near-midpoint values), stored them with image_store ... dmask into a 32x4 linear 2D T# of that format, one value per component, and read the raw bits back through a UINT T# over the same memory:

format	values	values where the two rules differ	hardware = round(exact x * scale)	hardware = RTNE(fl32(x * scale))
8_UNORM	128	12	128	116
8_8_8_8_UNORM	128	12	128	116
8_8_8_8_SNORM	128	11	128	117
16_16_16_16_UNORM	128	10	128	118
16_16_UNORM (the test's two-channel swizzle 0x22c)	128	10	128	118

In every row where the rules differ, the hardware stored the level nearest to the exact product, for example 0x3f4e4e4e (exact product 205.49999...) gives 205 in 8_UNORM, and -0x1.7cf9f4p-1 (exact -94.5000004) gives -95 in 8_8_8_8_SNORM. Clamping, NaN to 0 and ±infinity to the end levels match the test's expectations. So the fma rule you described (e = fma(x, scale, -p) to break the .5 cases of fl32(x * scale)) is the one that matches RDNA 2.

My counts per format differ from @jotitadev's because I stored all four values of every slot into every format, where the test's own layout stores fewer components for the one- and two-channel formats. Desktop RDNA 2, not a PS5.

[tibzejoker]

Thank you @thuanquang, that settles it: RDNA 2 and Apple M2 both store the level nearest to the exact product.

Pushed 61340e2 on top of the first commit:

• the shader computes p = fl32(x * scale) and its fma residual e = fma(x, scale, -p) (both NoContraction). When p is exactly a .5, e > 0 rounds up, e < 0 rounds down, e == 0 keeps ties to even;
• the test now expects nearbyint of the product computed in double (exact for f32 x 16-bit scale). The first version fails it on your example: texel (10, 0) component 0 is 0xce, expected 0xcd.

Passes 3/3 on an RTX 4090 and an Intel UHD 770 with ANYPS5_NO_SHADER_CACHE=1. The 9 agc_driver_image_store|agc_driver_storage tests pass too. On the UHD 770 the residual came out 0 until the fma itself was decorated NoContraction. Title and body updated.

@LeandroLP23 your 860M run was on the first version; if you have a moment, a rerun of agc_driver_image_store_unorm_rounding on this head would be welcome.

AI-assisted: yes (Claude Code)

[LeandroLP23]

Rerun on 61340e2, AMD Radeon 860M (RDNA 3.5, 1002:1114), Windows 11, WinLibs GCC 15.2.0, Release, ANYPS5_NO_SHADER_CACHE=1, merged into main fb0dc11:

• agc_driver_image_store_unorm_rounding passes 3 of 3.
• The other 10 agc_driver_image_store* and agc_driver_storage* tests pass too.
• With SpirvImageEmitter.cpp from the first commit and the new test, it fails 3 of 3 on the same texel you quoted: image_store of 8_UNORM: texel (10, 0) component 0 is 0x000000ce, expected 0x000000cd. So on this driver the fma residual is what makes the difference.

AI-assisted: yes (Claude Code)

[tibzejoker]

Thank you @LeandroLP23, that is the check I hoped for: on RDNA 3.5 the new head passes and the first version's shader fails on the same texel. I added the 860M rerun to the GPU line.

AI-assisted: yes (Claude Code)

[thuanquang]

One more detail for the fma rule's e == 0 case, same GPU: the only exact ties these formats can hit are ±0.5 (x * scale = ±63.5, ±127.5, ±16383.5, ±32767.5), and the hardware stores 0.5 → 64 / -0.5 → -64 in 8_8_8_8_SNORM and ±16384 in 16_16_16_16_SNORM (and 128 / 32768 in the UNORM formats earlier). So ties don't go toward +infinity (-63.5 doesn't give -63); round-to-even, as in your rule, gives the hardware level for every tie these formats can produce. -1.0 stores -127 in SNORM8 and -32767 in SNORM16.

[tibzejoker]

Thanks @thuanquang, good to have the exact ties pinned down too. ±0.5 giving ±64 / ±16384 in SNORM and 128 / 32768 in UNORM is round-to-even, and -1.0 giving -127 / -32767 matches the clamp to [-1, 1]. So the head's rule agrees with RDNA 2 on every case measured.

AI-assisted: yes (Claude Code)

[jotitadev]

Apple GPU / MoltenVK rerun on 61340e2. The new rule matches what Apple stores, and the fma residual survives Metal fast math. agc_driver_image_store_unorm_rounding passes, the first commit's shader fails it on the texel you quoted, and the full ^agc suite is unchanged.

Machine: Apple M2 Max, macOS 27.2, x86-64 build under Rosetta. Drivers: MoltenVK 1.4.2 (Vulkan SDK 1.4.363.0) and MoltenVK main + SPIRV-Cross #2704. Head 61340e2, PR diff applied cleanly.

Tested: the tree is main a373e85 + mugurc's macOS series (ci/macos-series-2) + the PRs needed to build on macOS (#6667, #6668, #7524, #6969, and two files of #7332), not plain main (it doesn't build on macOS yet). Applied together in the same build: #7666, #7811, #7799 and #7803, plus a test-only patch that lets nine AGC fixtures load the Vulkan loader on macOS (branch jotitadev:fix/agc-tests-vulkan-loader-macos). Base on that tree: 40 ^agc failures with MoltenVK 1.4.2 and 26 with MoltenVK main + #2704, plus agc_driver_pooled_buffer_range and agc_performance_overlay, which only run because of the loader patch.

• With the PR: agc_driver_image_store_unorm_rounding passes 3 of 3 on both drivers, with MVK_CONFIG_FAST_MATH_ENABLED set to 0, 1 (always fast math) and 2.
• With SpirvImageEmitter.cpp from the first commit (3e54993) and the new test, it fails 3 of 3 on both drivers, and also with fast math forced: image_store of 8_UNORM: texel (10, 0) component 0 is 0x000000ce, expected 0x000000cd (level nearest to the exact product). That is the same texel as on the 860M.
• With all of this PR's non-test changes reverted, the new test passes on Apple. Without the shader rounding, Apple's own float-to-unorm/snorm conversion already stores the level nearest to the exact product (my earlier review). So on Apple the test catches a regression to the first commit's rule, not the removal of the rounding.
• Why fast math does not break the residual: in the MSL that MoltenVK generates (MVK_CONFIG_DEBUG=1, fast math forced), SPIRV-Cross turns the NoContraction multiply and subtract into its [[clang::optnone]] helpers spvFMul (fma(l, r, 0)) and spvFSub (fma(-1, r, l)). The residual is a plain fma(x, scale, -p). The compiler can't fold p back into the residual.
• Full ^agc suite with the stack: the base failures on each driver, nothing new and nothing gone.

GPUs tested: Apple M2 Max (MoltenVK 1.4.2; MoltenVK main + SPIRV-Cross #2704)

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

[tibzejoker]

Thank you @jotitadev. Good to know the residual survives forced fast math through SPIRV-Cross's optnone helpers. Noted too that on Apple the test guards against a return to the first rule rather than the removal of the rounding, since Metal already stores the nearest level. Added the rerun to the GPU line.

AI-assisted: yes (Claude Code)

[DavidAmsellem]

First AMD-on-Linux (RADV) run of this PR. Merged onto main 43a2abaf7 (clean merge), built in full and compared with the same main, on the same machine.

AMD Radeon RX 7900 XT (RADV NAVI31, gfx1100), Mesa 25.2.8, Linux Mint 22.3 (Ubuntu 24.04 base), kernel 7.0.0-38, GCC 13.3, Release.

Full suite (ANYPS5_REQUIRE_VULKAN=1 ctest -j8): 1637/1639 with this PR, against 1636/1638 on main. The failures are the same two on both, so this PR adds none: agc_driver_image_table_vouched_lookups (#7766, fixed by #7916) and agc_driver_guest_memory_lifetime (#7784). The new agc_driver_image_store_unorm_rounding ran (not skipped) and passed.

GPUs tested: AMD Radeon RX 7900 XT (RADV NAVI31, gfx1100), Mesa 25.2.8, Linux Mint 22.3 / kernel 7.0.0-38

AI-assisted: yes

[tibzejoker]

Thank you @DavidAmsellem, the first Linux/RADV run is a useful one to have: the new test ran and passed on gfx1100, and the full suite shows only the two failures main has too. Added to the GPU line.

AI-assisted: yes (Claude Code)