AGC: vkQuake's ray-traced shadow compute shader becomes 519,500 SPIR-V words; Intel ANV compiles it for minutes under GpuMutex and crashes (PPSA99010, Linux)

vkQuake (PPSA99010) with its default ray-traced shadows freezes on Linux with an Intel GPU. One compute shader of the title's run-time RADV port becomes a 519,500-word SPIR-V module. Intel's compiler works on it for minutes while the AGC queue thread holds GpuMutex, so the title's main thread blocks, and after about 5 minutes Mesa itself segfaults.

Setup: main 7e181fc, Arch, GCC 16.2.1 Release, Intel Core Ultra X7 358H (Arc B390, Mesa 26.2.2 ANV, host subgroup 32), relinked with --to-intel, Quake 1.06 shareware id1/pak0.pak (MD5 5906e5998fc3d896ddaf5e6a62e03abb), Xwayland. Same title and data as the Windows reports in #1578.

What happens

With APS5_PROFILE_DRAW=1 the last line is [dispatch] creating a pipeline for 519500 SPIR-V words (program 0x20bf7f400). Nothing follows: in a 10-minute run the process segfaults about 5 minutes later inside libvulkan_intel.so (kernel: segfault at 40 ... in libvulkan_intel.so). The kernel log shows no GPU hang or reset.
Interrupted in gdb while stuck, the AGC queue thread is in Driver::run → execute → dispatch → VulkanDevice::Dispatch → libvulkan_intel.so (deep recursive frames), under the GpuMutex that Dispatch.cpp takes before localDevice->Dispatch. The title's main thread waits for it in sceKernelMunmap → DoMunmap → GuestAllocationsUnmap → WaitForLeasesAndImports → GpuMutexType::lock(). So the whole title stops, not just the frame.
With r_rtshadows 0 the same build plays the demo loop (2 of 2 runs of 120 s).

Where the size comes from

The program is 2,748 bytes of ISA (pc 0 to 0x2af0). Counts from APS5_DUMP_IR=20bf7f400 and from the SPIR-V in an APS5_DUMP_COMPUTE_PIPELINES capture:

Stage	Size
IR after `translate`	~60,000 values (12,397 `BitwiseAnd32`, 8,062 `SelectU32`, 5,044 `IEqual32`, 4,579 `GetVectorRegister`)
IR after `resources`	~38,000 values in 185 blocks
SPIR-V	519,500 words, 110,811 instructions, 6,074 `OpLabel`; top opcodes `OpBitwiseAnd` 22,722, `OpSelect` 15,075, `OpBitcast` 10,482, `OpIEqual` 9,591
`spirv-opt -O` of that SPIR-V	461,429 words (−11%)

Three things stack up:

f32 denormal flushing. The shader's FLOAT_MODE flushes f32 denormals, so f32DenormalFlushFor wraps every add, mul, fma, min/max, compare and conversion in the exact flush emulation (flushF32Denormal, flushTinyProduct). As a local experiment only, I made f32DenormalFlushFor return 0: the module drops to 340,562 words (−34%), and the final IR to 21,200 values. It still did not compile within 120 s.
Two lanes per invocation. The module has LocalSize 32 1 1, and the SPIR-V repeats each operation for a pair of values (%45932/%45933, ...), consistent with a wave64 program on a 32-wide host subgroup. That roughly doubles everything after the IR.
IEEE min/max and NaN rules. Runs like OpFOrdLessThan + 5 OpSelect + exponent masks per v_min/max_f32, plus > 0x7f800000 → 0xffc00000 canonicalisation.

The closed #3842 (Killzone: Liberation, 649,515 words, ANV stuck in brw_opt_register_coalesce) had the same symptom from a different cause (relative VGPR select chains, fixed by the vector window). #4042 proposes cheaper special-value rules for packed f16; this shader is f32.

What I don't know

Whether NVIDIA or AMD drivers compile this module in reasonable time: on NVIDIA the title stops earlier on this shader family with the Shader float mode (IEEE_MODE, DX10_CLAMP, FLOAT_MODE) is per shader; the recompiler assumes one #981 s_round_mode error.
Whether the denormal emulation could use SPIR-V DenormFlushToZero where the device reports shaderDenormFlushToZeroFloat32, given the before/after-rounding distinction flushTinyProduct models.
Whether building a pipeline should happen without GpuMutex held, so a slow driver compile stalls one queue instead of every munmap.

I have the 2 MB capture.aps5cp (replays with agc_compute_pipeline_replay built with ANYPS5_ENABLE_SPIRV_TOOLS=ON) and the IR dump, and can attach them or rerun anything.

I ran PR #4948's agc_redundant_value_elimination test locally on Linux (GCC 12.2, Release); it passes (1/1). That validates the eliminator tests, but neither that test nor the PR's hosted checks exercise the 519,500-word vkQuake shader.

Since you mentioned having the capture.aps5cp, could you attach it here or share a link? We can then measure the IR/SPIR-V size and pipeline preparation time with and without #4948 on that exact capture.

Another title hits this on Intel: Astro Bot (PPSA21559 v01.018) on main 8cf6a0b, Arc Pro B50 (Battlemage), Mesa 26.2.2 ANV, Fedora 44. Details in #357.

ANV pipeline creation in agc_compute_pipeline_replay, Mesa shader cache disabled:

SPIR-V words	`OpLabel`	`OpPhi`	`OpLoopMerge`	Pipeline creation
151,691	4,005	820	0	4.9 s
425,609	522	50	0	15.8 s
621,969	6,789	4,210	8	204 s
408,316	3,152	2,179	8	763 s, 13.9 GB peak RSS

Two data points that may help whoever picks the fix:

Neither size nor block count alone predicts the cost. The two slow modules are the only ones of the eight captured with 8 loops and thousands of OpPhi; all eight share the same top opcodes (OpBitwiseAnd, OpSelect, OpBitcast, OpIEqual).
Compiling that one module peaks at 13.9 GB RSS. Inside a title already holding about 17 GB, that is what runs the host out of memory, before the crash at about 5 minutes you saw.

I'll measure #3655, #4948 and #2837 on these captures and report.

AI-assisted: yes

Thanks for the Astro Bot measurements. The two very slow modules having thousands of OpPhi nodes and loops is a useful lead, and the 13.9 GB peak makes the memory pressure concrete. Please share the #3655/#4948/#2837 comparison when you have it; especially useful would be whether any of those changes reduce compile time or peak RSS for the exact vkQuake capture, not only the module size.

Follow-up with measurements on the Astro Bot program that stalls (0x5005f6100, see my comment above), to see which lever matters on ANV.

Method. One title run with APS5_DUMP_SHADERS=1 saved the recompile request. Each variant below is main 8cf6a0b with the pull request merged, and agc_shader_replay --spv regenerates the module. That module is spliced into the title's APS5_DUMP_COMPUTE_PIPELINES capture (same layouts) and timed with agc_compute_pipeline_replay, Mesa shader cache disabled. The offline module uses the tool's default target, so it is smaller than the title's (362,818 words against 408,316). It has the same structure, though (2,181 OpPhi, 8 loops, 256 calls), and the same cost on ANV: 831 s and 13.9 GB, against 763 s and 13.9 GB for the title's module.

Variant	SPIR-V words	ANV pipeline creation	Peak RSS
`main`	362,818	831 s	13.9 GB
#3655	362,019 (−0.2%)
#4948	362,818 (unchanged)
#2837	363,054
#4539	343,930 (−5%)
#4490 / #4737 / #5086	360,655 / 359,479 / 362,818
local experiment: `f32DenormalFlushFor` returns 0	277,808 (−23%)	183 s	6.6 GB

The other three stalling or slow programs (0x50064fc00, 0x555f4f500, 0x5007a9500) change in the same proportions: −23 to −36% without the flush, −5 to −10% with #4539.

Where the size is. In the main module, main itself is 338,446 words. The six BDA helpers are about 3,000 words of bodies, called 256 times: 205 record_bda_fault, 23 read_bda_span, 19 probe_bda_pointer, 4 get_bda_pointer, 4 read_bda_dword_bytes, 1 load_bda_span. Even inlined, that is about 50,000 words. main holds 6 loops of the guest program and 2,160 OpPhi. So for this program the BDA lookup changes cannot move the compile time much, and the f32 denormal flush emulation is the largest single factor: removing it cuts compile time by 4.5x and peak memory by half. Even then 183 s per pipeline is too slow for loading. Without the flush the top opcodes are still OpSelect 8,665, OpBitcast 7,522 and OpBitwiseAnd 4,208.

The experiment is not a fix proposal: dropping the flush changes results, and I can't verify a native DenormFlushToZero path against RDNA without AMD hardware. I can rerun this setup on any branch that changes the recompiler and report the ANV time.

AI-assisted: yes

Thanks for the controlled measurements. The comparison points to denormal flushing as the largest cost here, but disabling it changes results, so I don’t see a safe AnyPS5 change from these numbers alone. A native or otherwise semantics-preserving lowering would need correctness evidence on AMD before I’d propose it.

NVIDIA data for the open question above (whether NVIDIA compiles these modules in reasonable time). It no longer stops on #981 there, and it compiles them, but slowly.

Setup: main d167000, Ubuntu 24.04, GCC 13.3, Release, RTX 4070 Ti (NVIDIA 580.142), relinked with --to-intel, Quake 1.06 shareware id1/pak0.pak (MD5 5906e5998fc3d896ddaf5e6a62e03abb), default settings, APS5_PROFILE_DRAW=1. An earlier run on main 7a0c838 with #5464, #3925, #5523 and #5550 merged gave 30.2 s, 31.2 s and 32.1 s for the same three programs.

Program	SPIR-V words	Pipeline creation
`0x21174f400` (192x1x1)	417,134	31.0 s
`0x206d5d800` (4x1x1)	151,540	32.4 s
`0x206d61000` (4x1x1)	151,514	30.9 s

The three are created one after another, so the queue is busy for about 94 s ([gpu] worker at 111 s: dispatch 107.2 s). Interrupted in gdb meanwhile, the AGC queue thread is in VulkanDevice::dispatch → libnvidia-glcore → libnvidia-gpucomp.
The title presents 12 frames (its loading plaque), all before the first of these pipelines is ready, and none after. A few seconds after the third is created, it exits with status 0 and no message (114 s), before the demo loop draws.
Peak RSS of the whole process is 4.0 GiB, so on NVIDIA the cost here is time, not memory like the 13.9 GB ANV needed for Astro Bot's module.

I haven't tried r_rtshadows 0 here.

AI-assisted: yes

Correctness data for the native lowering @MarcZX100 asked about, from Intel, and a third title.

Intel's DenormFlushToZero gives the bits #1850 measured on RDNA2. Setup: Fedora 44, Intel Arc (Meteor Lake, Mesa 26.2.3 ANV, shaderDenormFlushToZeroFloat32 true, denormBehaviorIndependence 32-bit only), a standalone Vulkan program outside AnyPS5. Each operation runs as its own compute shader with DenormFlushToZero 32 and SignedZeroInfNanPreserve 32, over 217,600 rows: 26 values (8 denormals of both signs, zeros, min normal, 1, 0x3f7fffff, max, infinities, quiet and signaling NaN) crossed three ways, plus 200,000 random rows of products, sums and differences around 2^-126. The reference applies #1850's rules: inputs and result flushed to a zero of the same sign, and tininess before rounding for products. NaN results are compared as NaN, everything else bit for bit.

Operation	Mismatches with `DenormFlushToZero`	with `DenormPreserve`
`OpFAdd`, `OpFSub`	0	54,160 / 103,373
`OpFMul` (tiny products included)	0	3,848
`Fma`, zero addend	0
`Fma`, nonzero addend	2	12,072 (all fma rows)
`FMin`, `FMax`	0	32,563 / 29,085
`Floor`, `Fract`	0	26,859 / 56,084
`OpFOrdLessThan`, `OpFOrdEqual`	0	1,144 / 2,288
`OpConvertFToS`	0	0

The two Fma rows are the gap #1850 left open: 0x80800000 * 0x3f7fffff + 0x00ffffff, whose exact sum lies just under 2^-126 and rounds up to it. The Arc gives 0x00000000 (tininess before rounding), while the reference and the current emulation give 0x00800000. Without an AMD run I can't say which RDNA does. ANV with no execution mode at all flushes the same way.

What it saves, and what it doesn't. A local switch that leaves f32DenormalFlushFor at 0 and declares DenormFlushToZero 32 instead, on main f265cd55. With it, the 48 compute shaders Stray (PPSA02101) dispatches up to its second scene go from 2,171,795 to 1,650,246 SPIR-V words (−24%). Its largest, program 0x2010cf0000 (6,060 code words, 8x8x1 threads, wave64), goes from 624,132 to 436,504 (−30%). That one is what freezes Stray on Intel after the calibration screen. Neither version of it compiles here, though: created alone with vkCreateComputePipelines (required subgroup size 32, Mesa cache off), both were still compiling after more than 10 minutes, past 13 GB of RSS, when I stopped them. It is one function with 907 blocks, 42 OpPhi, no loops or calls, and about 90,000 instructions. The pair repetition of a wave64 program on a 32-wide subgroup and the min/max NaN rules account for most of the rest.

Proposal. I'd open a PR that uses the hardware flush only where it is shown to match: the driver checks, once per device, the rows above for each class of flushing instruction against the emulation's rules, and lowers a class through DenormFlushToZero 32 only if every row matches. Drivers without the property (NVIDIA) or with any mismatch keep today's emulation, and the fma with a nonzero addend stays emulated everywhere. That keeps results identical to today on every driver while making the modules about a quarter smaller where the check passes. It doesn't by itself make Stray's or vkQuake's largest shader compile on ANV; that needs the wave64 pairs to shrink too. Does this sound acceptable before I write it?

AI-assisted: yes.

The per-device probe is a useful way to gate a host-specific optimization, but one SPIR-V constraint needs to be part of the design: DenormFlushToZero 32 is an entry-point execution mode, and execution modes apply to the entry point’s whole static call graph. It cannot be selected only for add/mul/min/max while leaving a nonzero-addend Fma in the same entry point under the old behavior. Since your matrix reports two Fma mismatches, checking the mode once and retaining the current Fma lowering would not by itself isolate that mismatch.

I would make the mixed-operation case explicit in the regression: one shader containing a verified-safe operation and the nonzero-addend Fma counterexample, compared against the current emulator semantics. The SPIR-V spec describes the mode as applying to all operations of the target width in the entry point call graph: https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html#Execution_Mode. Vulkan’s shaderDenormFlushToZeroFloat32 property permits the mode; it does not establish that its result matches the guest behavior, so your per-device semantic probe remains important.

@ThatsLeo

RTX 5080 data for the per-device check @ThatsLeo proposed and for @MarcZX100's point that a mode covers the whole entry point. The driver offers neither f32 denormal mode. With no f32 mode it already flushes f32 denormals in most ops, as @SleglJan found on an RTX 3080 (#2523 (comment)). Which ops it flushes changes, though, when the module also declares DenormPreserve 16, which main does on NVIDIA.

Properties. From vulkaninfo on an RTX 5080 (Blackwell), Windows 11, NVIDIA 616.92:

`VkPhysicalDeviceFloatControlsProperties`	f16	f32	f64
`shaderDenormFlushToZero`	false	false	false
`shaderDenormPreserve`	true	false	false
`shaderSignedZeroInfNanPreserve`	true	true	true
`shaderRoundingModeRTE`	true	true	true
`shaderRoundingModeRTZ`	false	true	true

denormBehaviorIndependence and roundingModeIndependence are both ALL. The f32 denormal properties are the same as on @kr4y4's RTX 4070 Laptop (#981 (comment)) and @SleglJan's RTX 3080 (#981 (comment)).

Main's modes. For f32, main declares only SignedZeroInfNanPreserve 32 (SpirvModuleSetup.cpp#L152). On NVIDIA it adds DenormPreserve 16 in two cases:

when a shader unpacks f16 (SpirvModuleEmitter.cpp#L1058-L1059, enabled by VulkanDevice.cpp#L1218-L1219);
together with RoundingModeRTE 16 and SignedZeroInfNanPreserve 16, when it converts f32 to f16 or does packed f16 math (SpirvAluEmitterMath.cpp#L1420-L1430).

Probe. One standalone compute shader: one entry point, a switch over the ops, NoContraction on the arithmetic results (main's Exact adds it to Fma). It ran over 582,402 rows: 16 hand-picked rows, a 26-value cross product (no NaN inputs) and 560,000 random rows with exponents near 2^-126. I compared each result bit for bit against IEEE and against a flush rule. Under the rule, inputs and denormal results become a zero of the same sign, and a mul or fma result is tiny if it is below 2^-126 after rounding to 24 bits.

Ops (rows)	Only `SignedZeroInfNanPreserve 32`	Plus `DenormPreserve 16`
`OpFAdd`, `OpFSub` (100,678 / 100,677)	flushed: 0 mismatches with the rule	kept: 0 mismatches with IEEE
`OpFMul` (100,679)	flushed: 0 mismatches with the rule	kept: 0 mismatches with IEEE
GLSL `Fma` (137,580)	flushed: 0 mismatches with the rule	still flushed: 0 mismatches with the rule
`FMin`, `FMax` (20,677 each)	flushed; the only differences from the rule are the sign of a zero result	still flushed, same
`OpFOrdLessThan`, `OpFOrdEqual` (20,677 each)	flushed: 0 mismatches with the rule	kept: 0 mismatches with IEEE
`Floor` (20,027)	kept: `floor(0x80000001)` = `0xbf800000`	kept
`Fract` (20,027)	neither: `fract(0x00000001)` = `0`, `fract(0x80000001)` = `0x3f800000`	kept: 0 mismatches with IEEE

A tiny-before-rounding rule differs from the card in 9 mul and 81 fma rows. NoContraction made no difference, and repeat runs of these modules gave identical bits (two to four runs each). Adding main's other two f16 modes, RoundingModeRTE 16 and SignedZeroInfNanPreserve 16, gave the same bits as DenormPreserve 16 alone (one run).

Without the f16 mode, the ops that follow the rule match the classes whose flush @SleglJan's opt-in APS5_HOST_F32_FTZ experiment drops. With it, add, sub, mul and the compares keep f32 denormals. So a per-device probe should include the f16 modes main may add to the same entry point: on this card they change the f32 results.

The counterexample. On the row from @ThatsLeo's matrix (#4977 (comment)), Fma(0x80800000, 0x3f7fffff, 0x00ffffff) returns 0x00000000 in every variant, the same as the Arc with DenormFlushToZero.

For a nonzero addend, flushTinyProduct keeps the driver's Fma result, because the tiny test is ANDed with addend == 0 (OperandAccess.cpp#L163-L171). writeOperand then only flushes a denormal result (#L252). So, following that path, main's v_fma_f32 gives 0x00000000 for this row on this card, and 0x00800000 on a driver whose Fma keeps denormals. I did not run main's v_fma_f32 itself. For @MarcZX100's mixed-operation regression (#4977 (comment)), the current emulator's expected value for that row therefore already depends on the host.

Not measured. I did not measure SPIR-V size or compile time with and without the flush emulation, because no capture of the vkQuake, Astro Bot or Stray programs is attached here. The closest NVIDIA numbers are @SleglJan's in the #2523 comment above. On the RTX 3080, 007's largest module (folded with spirv-opt) went from 4.90M words and 130.4 s of pipeline creation to 2.83M words and 72.2 s with the flush and the plain-multiply tiny-product check stripped for the classes NVIDIA flushed. If someone attaches a capture.aps5cp, I can time it with agc_compute_pipeline_replay on this card. I can also attach the probe (SPIR-V assembly and C++ harness).

AI-assisted: written with Claude Code; I checked the property values in both vulkaninfo and the probe's own query, the code paths at main e34c479 and the comments linked above, and ran vulkaninfo and the probe (four module variants, all but one run more than once) on the RTX 5080.

@nikowolf123

A correction to my earlier comment, and the mixed-operation check @MarcZX100 asked for, on the Arc (Meteor Lake, Mesa 26.2.3 ANV) at main e154f5658.

Correction. I wrote that the current emulation gives 0x00800000 for 0x80800000 * 0x3f7fffff + 0x00ffffff. On ANV it doesn't. @nikowolf123 was right: with a nonzero addend, main keeps the driver's Fma result. ANV flushes that row even with no execution mode, so main already returns 0x00000000 there.

One shader, every operation. agc_driver_f32_denormal_flush runs 19 flushing instructions in one compute entry point:

v_add, v_mul, v_fma, v_fmac, v_min, v_max3, v_med3, v_floor, v_frexp_*, v_cmp_lt, v_div_fixup, v_mullit and others;
under every float mode the test covers.

I put the counterexample in its last row and ran it three ways with a local switch:

Module	Mismatches with the test's expected values	The counterexample's `v_fma_f32`
`main` (emulation)	0	`0x00000000`
`DenormFlushToZero 32` declared, emulation kept	0	`0x00000000`
`DenormFlushToZero 32` declared, `f32DenormalFlushFor` returning 0 for every instruction	564	`0x00000000`

Mode declared, emulation kept. It also passes the other 23 float tests and the whole libSceAgcDriver suite: 554 of 556, and the two failures fail without the mode too. So on ANV the mode doesn't change the instructions that stay emulated in the same entry point, including that fma row.
Every software flush dropped. That is not an option. The hardware flush doesn't cover these instructions:
- v_min, v_max3 and v_med3, lowered with the NaN rules;
- v_frexp_*, v_mullit, v_mul_legacy, v_mad_legacy and v_mac_legacy;
- v_div_fixup and v_cmp_lt (one row in each mode);
- v_mul and v_fma_mix under the non-default rounding modes.
It also accepts a FLOAT_MODE that the emulation refuses.

What a probe could drop. Only the classes the mixed test leaves at 0 mismatches could drop their software flush, and only in the default rounding mode: v_add/v_sub, v_mul, v_fma/v_fmac and v_floor/v_ceil. v_cmp_lt differed in one row per mode. Whatever the device does with the mode, the probe has to run the recompiler's own lowering in one module, not each operation alone; my standalone probe showed 0 mismatches for FMin and the compares, but the recompiler doesn't lower v_min to FMin.

AI-assisted: yes

Navigation Menu

Repository navigation

AGC: vkQuake's ray-traced shadow compute shader becomes 519,500 SPIR-V words; Intel ANV compiles it for minutes under GpuMutex and crashes (PPSA99010, Linux) #4977

What happens

Where the size comes from

What I don't know

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

AGC: vkQuake's ray-traced shadow compute shader becomes 519,500 SPIR-V words; Intel ANV compiles it for minutes under GpuMutex and crashes (PPSA99010, Linux) #4977

Description

What happens

Where the size comes from

What I don't know

Activity

MarcZX10 commented on Oct 13, 2026

Faffu commented on Oct 13, 2026

MarcZX10 commented on Oct 13, 2026

Faffu commented on Oct 13, 2026

MarcZX10 commented on Oct 13, 2026

milescoviello commented on Oct 14, 2026

ThatsLeo commented on Oct 14, 2026

MarcZX10 commented on Oct 14, 2026

nikowolf123 commented on Oct 14, 2026

ThatsLeo commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions