Skip to content

AGC: Borderlands 4 (PPSA01494) segfaults in NVIDIA 595.99's shader compiler on a valid recompiled compute shader #6294

Closed
Closed
AGC: Borderlands 4 (PPSA01494) segfaults in NVIDIA 595.99's shader compiler on a valid recompiled compute shader#6294

Description

@milescoviello
milescoviello
opened on Oct 15, 2026

Borderlands 4 (PPSA01494, 01.100.002) segfaults inside NVIDIA's shader compiler while AnyPS5 creates one of its compute pipelines. Linux (Ubuntu 24.04), RTX 4070 Ti, NVIDIA 595.99.02, GCC 13.3, Release, relinked with --to-intel, pad input from 8 s.

What happens

Five 600 s runs on main d92a8da, db81992 and 721db55 (one of them with #6160 and #6094 merged) all end with EXIT-139 between 220 s and 381 s, with no AnyPS5 message. The core dump's stack:

#0  libnvidia-gpucomp.so.595.99.02
...
#13 libnvidia-glvkspirv.so.595.99.02
#14 _nv002nvvm () from libnvidia-glvkspirv.so.595.99.02
#15-#22 libnvidia-glcore.so.595.99.02
#23 AgcDriver::VulkanDevice::computePipeline(...)
#24 AgcDriver::VulkanDevice::PrepareDispatch(...)

Outside the title

With APS5_DUMP_COMPUTE_PIPELINES (its 128-entry limit is reached at ~204 s, so I raised it in a throwaway build), the last capture before the crash replays with agc_compute_pipeline_replay and segfaults the same way, 3 of 3 runs; the four captures before it replay without error. Its module (28,744 words, SPIR-V 1.4, LocalSize 32 1 1, a wave64 program on 32-wide subgroups) passes spirv-val (SPIRV-Tools v2025.1).

Changing the module and replaying:

  • Replacing its 64 OpGroupNonUniformBallot ... %true with OpUndef: no crash, 3 of 3. Keeping only one of them (the rest OpUndef) still crashes.
  • Still crashes: without SignedZeroInfNanPreserve, without the OpGroupNonUniformShuffles, after spirv-opt -O, with the ballots taking a non-constant true (SubgroupLocalInvocationId != 0xffffffff), and with each lo/hi pair of identical Ballot(true) merged into one.

spirv-reduce and a manual pass leave the module below, which still crashes 3 of 3. Its OpUndefs come from the reduction; the original has values there. What is left is the recompiler's lane-mask pattern: two Ballot(true) words packed into a uvec4, a lane bit test, a shuffle, an unrolled clear-lowest-bit chain, and a ballot of the result.

Reduced module (spirv-dis --raw-id) 
; SPIR-V
               OpCapability Sampled1D
               OpCapability Shader
               OpCapability GroupNonUniform
               OpCapability GroupNonUniformBallot
               OpCapability GroupNonUniformShuffle
               OpCapability SignedZeroInfNanPreserve
               OpCapability Int64
               OpExtension "SPV_KHR_float_controls"
          %1 = OpExtInstImport "GLSL.std.450"
               OpMemoryModel Logical GLSL450
               OpEntryPoint GLCompute %2 "main" %3
               OpExecutionMode %2 SignedZeroInfNanPreserve 32
               OpExecutionMode %2 LocalSize 32 1 1
               OpDecorate %3 BuiltIn SubgroupLocalInvocationId
          %4 = OpTypeInt 32 0
          %5 = OpTypePointer Input %4
          %3 = OpVariable %5 Input
          %7 = OpConstant %4 3
          %8 = OpTypeFloat 32
         %13 = OpTypeImage %4 2D 0 0 0 1 Unknown
         %16 = OpTypeInt 32 1
         %20 = OpTypeVoid
         %21 = OpConstant %4 0
         %22 = OpConstant %4 1
         %23 = OpTypeBool
         %30 = OpTypeFunction %20
         %35 = OpConstant %4 16
         %42 = OpConstant %4 64
         %49 = OpTypeInt 64 0
         %57 = OpConstantTrue %23
         %58 = OpTypeVector %4 4
         %61 = OpTypeVector %4 2
        %346 = OpConstant %4 256
        %347 = OpTypeArray %4 %346
        %353 = OpConstant %4 4294967280
        %354 = OpConstant %4 15
        %588 = OpUndef %23
        %589 = OpUndef %4
        %590 = OpUndef %58
        %591 = OpUndef %61
        %592 = OpUndef %49
        %593 = OpUndef %13
        %594 = OpUndef %16
          %2 = OpFunction %20 None %30
        %595 = OpLabel
               OpBranch %596
        %596 = OpLabel
        %597 = OpUndef %4
        %598 = OpUndef %4
        %599 = OpUndef %4
        %600 = OpUndef %4
        %601 = OpUndef %4
        %602 = OpBitwiseAnd %4 %589 %353
        %603 = OpBitwiseXor %4 %602 %35
        %604 = OpUndef %4
        %605 = OpBitwiseAnd %4 %604 %354
        %606 = OpBitwiseOr %4 %603 %605
        %607 = OpGroupNonUniformShuffle %4 %7 %589 %589
        %608 = OpGroupNonUniformShuffle %4 %7 %589 %589
        %609 = OpUndef %23
        %610 = OpSelect %4 %588 %608 %607
        %611 = OpGroupNonUniformBallot %58 %7 %57
        %612 = OpGroupNonUniformBallot %58 %7 %57
        %613 = OpCompositeExtract %4 %611 0
        %614 = OpCompositeExtract %4 %612 0
        %615 = OpCompositeConstruct %58 %613 %614 %21 %21
        %616 = OpCompositeExtract %4 %615 0
        %617 = OpCompositeExtract %4 %615 1
        %618 = OpUndef %23
        %619 = OpSelect %4 %588 %617 %616
        %620 = OpUndef %4
        %621 = OpShiftLeftLogical %4 %22 %589
        %622 = OpBitwiseAnd %4 %619 %621
        %623 = OpINotEqual %23 %622 %21
        %624 = OpULessThan %23 %606 %42
        %625 = OpLogicalAnd %23 %623 %624
        %626 = OpSelect %4 %625 %610 %21
        %627 = OpULessThan %23 %589 %589
        %628 = OpSelect %4 %627 %599 %626
        %629 = OpINotEqual %23 %589 %21
        %630 = OpUndef %4
        %631 = OpUndef %23
        %632 = OpUndef %4
        %633 = OpSelect %4 %629 %632 %628
        %634 = OpUndef %4
        %635 = OpUndef %4
        %636 = OpSelect %4 %629 %635 %633
        %637 = OpUndef %4
        %638 = OpSelect %4 %629 %637 %636
        %639 = OpUndef %4
        %640 = OpSelect %4 %629 %639 %638
        %641 = OpUndef %4
        %642 = OpUndef %23
        %643 = OpUndef %4
        %644 = OpINotEqual %23 %21 %640
        %645 = OpIAdd %4 %354 %640
        %646 = OpSelect %4 %644 %645 %643
        %647 = OpBitwiseAnd %4 %640 %646
        %648 = OpSelect %4 %644 %647 %640
        %649 = OpINotEqual %23 %21 %648
        %650 = OpLogicalAnd %23 %644 %649
        %651 = OpIAdd %4 %354 %648
        %652 = OpSelect %4 %650 %651 %646
        %653 = OpUndef %4
        %654 = OpSelect %4 %650 %653 %648
        %655 = OpINotEqual %23 %21 %654
        %656 = OpLogicalAnd %23 %650 %655
        %657 = OpUndef %4
        %658 = OpSelect %4 %656 %657 %652
        %659 = OpBitwiseAnd %4 %654 %658
        %660 = OpSelect %4 %656 %659 %654
        %661 = OpINotEqual %23 %21 %660
        %662 = OpLogicalAnd %23 %656 %661
        %663 = OpLogicalNot %23 %662
        %664 = OpGroupNonUniformBallot %58 %7 %663
               OpBranch %665
        %665 = OpLabel
               OpReturn
               OpFunctionEnd

I haven't found a rewrite that keeps the shader's meaning and avoids it. Does anyone see it on another NVIDIA driver (Windows, or Linux 580.x)? #3355 was a different construct on the same compiler.

AI-assisted: yes

Activity

  1. jotitadev commented on Oct 17, 2026

    @jotitadev
    jotitadev
    on Oct 17, 2026
    Contributor

    Reproduces on Ampere with the same driver. RTX 3070 (GA104), NVIDIA 595.99.02, Ubuntu 26.04 (kernel 7.0.0-38). I assembled the reduced module from the report with spirv-as --target-env vulkan1.3 (SPIRV-Tools from 3rdparty/; spirv-val passes) and created one compute pipeline from it in a small standalone Vulkan program, not AnyPS5 code: an empty pipeline layout, no pipeline cache, __GL_SHADER_DISK_CACHE=0, one fresh process per run. The process segfaults inside vkCreateComputePipelines.

    Pipeline creation RTX 3070, 595.99.02 lavapipe (Mesa 26.0.3)
    no flags segfault 5/5 3/3 created
    REQUIRE_FULL_SUBGROUPS + requiredSubgroupSize 32 + DISPATCH_BASE (what AnyPS5 passes for a spanned wave64 program) segfault 5/5 3/3 created
    VK_PIPELINE_CREATE_DISABLE_OPTIMIZATION_BIT segfault 5/5

    So it is neither Ada-only nor caused by the pipeline flags, and turning off the driver's optimization doesn't avoid it.

    Smaller module without OpUndef. I replaced every OpUndef in the function with a value computed from SubgroupLocalInvocationId, then removed instructions greedily, replacing each value with that invocation-derived value while the crash remained. I also dropped the capabilities and types that became unused. The module below segfaults 5/5, and 5/5 with DISABLE_OPTIMIZATION. lavapipe creates it 3/3. All of it is dead code: nothing is stored.

    Module (spirv-dis --raw-id) 
    ; SPIR-V
; Version: 1.6
; Generator: Khronos SPIR-V Tools Assembler; 0
; Bound: 66
; Schema: 0
               OpCapability Shader
               OpCapability GroupNonUniform
               OpCapability GroupNonUniformBallot
               OpMemoryModel Logical GLSL450
               OpEntryPoint GLCompute %2 "main" %3
               OpExecutionMode %2 LocalSize 32 1 1
               OpDecorate %3 BuiltIn SubgroupLocalInvocationId
          %4 = OpTypeInt 32 0
          %5 = OpTypePointer Input %4
          %3 = OpVariable %5 Input
          %6 = OpConstant %4 3
         %10 = OpTypeVoid
         %11 = OpConstant %4 0
         %12 = OpConstant %4 1
         %13 = OpTypeBool
         %14 = OpTypeFunction %10
         %15 = OpConstant %4 16
         %16 = OpConstant %4 64
         %18 = OpConstantTrue %13
         %19 = OpTypeVector %4 4
         %24 = OpConstant %4 15
          %2 = OpFunction %10 None %14
         %30 = OpLabel
         %31 = OpLoad %4 %3
         %32 = OpIAdd %4 %31 %12
         %33 = OpULessThan %13 %31 %15
         %34 = OpBitwiseXor %4 %31 %15
               OpBranch %35
         %35 = OpLabel
         %36 = OpGroupNonUniformBallot %19 %6 %18
         %37 = OpCompositeExtract %4 %36 0
         %38 = OpCompositeConstruct %19 %32 %37 %11 %11
         %39 = OpCompositeExtract %4 %38 1
         %40 = OpSelect %4 %33 %39 %32
         %41 = OpBitwiseAnd %4 %40 %32
         %42 = OpINotEqual %13 %41 %11
         %43 = OpULessThan %13 %32 %16
         %44 = OpLogicalAnd %13 %42 %43
         %45 = OpSelect %4 %44 %32 %11
         %46 = OpSelect %4 %33 %32 %45
         %47 = OpSelect %4 %33 %32 %46
         %48 = OpSelect %4 %33 %32 %47
         %49 = OpSelect %4 %33 %32 %48
         %50 = OpSelect %4 %33 %32 %49
         %51 = OpINotEqual %13 %11 %50
         %52 = OpIAdd %4 %24 %50
         %53 = OpSelect %4 %51 %52 %32
         %54 = OpBitwiseAnd %4 %50 %53
         %55 = OpSelect %4 %51 %54 %50
         %56 = OpSelect %4 %33 %32 %53
         %57 = OpSelect %4 %33 %32 %55
         %58 = OpSelect %4 %33 %32 %56
         %59 = OpBitwiseAnd %4 %57 %58
         %60 = OpSelect %4 %33 %59 %57
         %61 = OpINotEqual %13 %11 %60
         %62 = OpLogicalAnd %13 %33 %61
         %63 = OpLogicalNot %13 %62
         %64 = OpGroupNonUniformBallot %19 %6 %63
               OpBranch %65
         %65 = OpLabel
               OpReturn
               OpFunctionEnd

    What it still needs, each change tried 3/3 on this module:

    • A ballot whose word feeds a lane bit test (%36 → %41/%42). If %36 is replaced by a non-ballot uvec4, the crash goes away. A ballot of a varying predicate (SubgroupLocalInvocationId < 16) instead of true still crashes.
    • The two clear-lowest-bit steps (%54, %59) and a ballot of the result (%64). A ballot of %61 or %62 still crashes. A ballot of an earlier value doesn't crash: the bit test (%42 or %44), or %51. Neither does replacing %64 with a plain OpSelect, or ballotting true there.
    • The shuffles are not needed. The second Ballot(true) of the lo/hi pair is not needed either.

    One rewrite that looks like a workaround but isn't. Replacing the bit test in the reduced module with OpGroupNonUniformBallotBitExtract (on %615, or on (%619, 0, 0, 0)) avoids the crash there, 3/3. But when the lane comes from SubgroupLocalInvocationId instead of OpUndef, BallotBitExtract((%619, 0, 0, 0), lane & 31) segfaults 5/5. Shift-and-mask, OpBitFieldUExtract and (mask & bit) == bit forms of the test still crash in the reduced module. In AnyPS5 the test comes from EmitBallotLaneActiveBool (SpirvSubgroup.cpp), so changing how that one function tests the bit doesn't look like an escape.

    Not tested: other driver versions (595.99.02 is the only one on this machine) and Windows.

    Written with AI assistance (Claude Code). I ran the tests above on my machine.

  2. milescoviello commented on Oct 18, 2026

    @milescoviello
    milescoviello
    on Oct 18, 2026
    ContributorAuthor

    Thanks @jotitadev, and sorry for not answering sooner. An Ampere reproduction on the same driver, and a reduction this far down, is very useful. It means the crash isn't specific to Ada, and that changing how EmitBallotLaneActiveBool tests the bit isn't an escape, so a workaround in AnyPS5 would have to avoid the ballot-of-a-bit-test pattern itself. I'll check whether a newer driver still crashes on your reduced module when one is available here.

    AI-assisted: yes (Claude Code)

  3. MarcZX10 commented on Oct 18, 2026

    @MarcZX10
    MarcZX10
    on Oct 18, 2026
    Contributor

    I traced the ballot-and-bit-test sequence back to the emitter. EmitSubgroupLaneActiveBool emits a ballot of true, then EmitBallotLaneActiveBool extracts the relevant word and tests the requested lane bit; the later ballot is a separate guest predicate. So the initial ballot is carrying the host subgroup's active-invocation mask, not just a redundant copy that can be replaced with an all-ones constant. In particular, any replacement has to preserve behavior for a partially populated subgroup as well as wave32/wave64 mapping.

    That also means #6644's same-predicate ballot reuse does not by itself remove this dependency: the minimized reproducer has one initial ballot and a distinct final ballot. I don't see a semantics-preserving rewrite to submit yet; a candidate needs an explicit active-mask replacement and a regression that proves it on partial subgroups, in addition to the NVIDIA pipeline-creation case.

    AI-assisted: yes

  4. milescoviello commented on Oct 18, 2026

    @milescoviello
    milescoviello
    on Oct 18, 2026
    ContributorAuthor

    @MarcZX10 thanks for tracing it to the emitter. That settles the workaround question for now. The first ballot carries the host subgroup's active mask, so replacing it needs an explicit active-mask replacement that's correct for partial subgroups and for both wave32 and wave64. A shortcut wouldn't do it. I won't propose a rewrite without a regression on partial subgroups, as you say. I'll leave this open as an NVIDIA driver issue, and retest jotitadev's reduced module when a newer driver is available on this machine.

    AI-assisted: yes (Claude Code)

  5. milescoviello commented on Oct 20, 2026

    @milescoviello
    milescoviello
    on Oct 20, 2026
    ContributorAuthor

    A negative result, so nobody else spends time on it. I tried computing the active mask without OpGroupNonUniformBallot %true. In min2 (the 47-instruction module), the two ballots of true became a subgroup OR of each lane's bit: OpGroupNonUniformBitwiseOr Reduce over lane < 32 ? 1 << lane : 0 for word 0, and the same for lanes 32 to 63 in word 1. That is correct for partial subgroups and for both subgroup sizes. NVIDIA 595.99.02 on an RTX 4070 Ti still crashes on it, 3 of 3 through agc_compute_pipeline_replay.

    What min2 needs in order to crash, 3 runs per variant:

    Variant of min2 Result
    as is segfault 3/3
    true-ballots replaced by the OR reduction segfault 3/3
    true-ballots replaced by plain values runs 3/3
    the two OpGroupNonUniformShuffle replaced by copies runs 3/3
    the last OpGroupNonUniformBallot (of the computed predicate, result unused) replaced by a constant runs 3/3

    So the crash needs a subgroup-derived value, fed through the shuffle-dependent OpSelect chain, into another ballot. Replacing one subgroup op by another doesn't avoid it. A workaround would have to remove the subgroup op behind the active-lane test altogether, or break the chain into the later ballot. I don't see a way to do that without knowing which lanes are active.

    AI-assisted: yes (Claude Code)

  6. milescoviello commented on Oct 21, 2026

    @milescoviello
    milescoviello
    on Oct 21, 2026
    ContributorAuthor

    Another data point, from an older driver branch on an older GPU. I created a pipeline from the reduced module (min2) in a standalone Vulkan program: no AnyPS5 code, an empty pipeline layout, no pipeline cache, __GL_SHADER_DISK_CACHE=0, one fresh process per run. The program is the same as jotitadev's test.

    GPU Driver vkCreateComputePipelines
    RTX 4070 Ti (Ada), Ubuntu 24.04 NVIDIA 595.99.02 segfault 3/3
    GT 1030 (Pascal), Windows 11 NVIDIA 582.78 (R580) returns VK_SUCCESS 3/3

    So the crash is not in every NVIDIA compiler. Either it came after the R580 branch, or it is in the Ampere/Ada code generation (jotitadev's RTX 3070 crashes as well). These two runs can't tell which, because both the driver and the GPU generation differ. R580 is the last branch that supports Pascal, so that card can't run a newer driver. The next check is a newer Linux driver on the 4070 Ti.

    AI-assisted: yes (Claude Code)

  7. milescoviello commented on Oct 21, 2026

    @milescoviello
    milescoviello
    on Oct 21, 2026
    ContributorAuthor

    NVIDIA 615.71.09 fixes it. This is the same RTX 4070 Ti, now on Ubuntu 26.04.1 with kernel 7.0.0, and the same standalone program and reduced module (min2) as in my last comment:

    GPU Driver vkCreateComputePipelines
    RTX 4070 Ti NVIDIA 595.99.02 segfault 3/3 (before)
    RTX 4070 Ti NVIDIA 615.71.09 VK_SUCCESS 3/3
    GT 1030 NVIDIA 582.78 (R580) VK_SUCCESS 3/3

    So the crash is in the 595 drivers and is fixed by 615. Users on 595.x should update. I'll confirm with a Borderlands 4 run past the old 220 to 426 s crash window, then close this as fixed in the driver.

    AI-assisted: yes (Claude Code)

  8. milescoviello commented on Oct 21, 2026

    @milescoviello
    milescoviello
    on Oct 21, 2026
    ContributorAuthor

    Correction to my last comment: 615.71.09 does not fully fix this. The reduced modules (min2 and the pass-1 reduced1) now create their pipelines. The full captured module (edb, 28,744 words) still segfaults 3 of 3 on 615.71.09, inside NVIDIA's compiler as before:

    #7  _nv002nvvm
...
#17 vkCreateComputePipelines

    That run used the same standalone program, which creates the pipeline with an empty pipeline layout. The full module declares descriptor bindings, so the layout doesn't match it, and that could matter. I'll replay the original capture with its recorded layout (agc_compute_pipeline_replay) on 615.71.09 to settle it. A Borderlands 4 run on 615.71.09 at a 12 GB cap also ended with EXIT-139 at 319 s, after UE's "GameThread timed out waiting for RenderThread", at its memory cap. That run doesn't show whether the compiler crash is still the cause. This issue stays open until the replay answers it.

    AI-assisted: yes (Claude Code)

  9. milescoviello commented on Oct 21, 2026

    @milescoviello
    milescoviello
    on Oct 21, 2026
    ContributorAuthor

    Settled with the original capture. agc_compute_pipeline_replay on compute-edb626b757e75707, the capture that segfaulted 3 of 3 on 595.99.02 (see the report), uses its recorded pipeline layout, requiredSubgroupSize 32 and pipeline flags. On 615.71.09 it creates the pipeline 3 of 3, each in about 146 ms (pipeline_create, fresh process, __GL_SHADER_DISK_CACHE=0). The capture recorded just before it, compute-19c1038e6c468f0e, also replays 3 of 3.

    So the full module's segfault in my last comment came from my standalone program's empty pipeline layout, which isn't valid for that module and isn't what Borderlands 4 creates. The title's own pipeline compiles on 615.71.09. That run still crashes the 615 compiler instead of failing cleanly, but it's invalid usage and outside AnyPS5.

    Closing this as fixed in NVIDIA's driver: update to 615.71.09 or newer. Borderlands 4's remaining stop on this machine is different. At a 12 GB cap it ends at 319 s after UE's "GameThread timed out waiting for RenderThread" watchdog, with the process at the cap, the same memory stop seen at 12 GB before.

    AI-assisted: yes (Claude Code)

  10. milescoviello
    closed this as completedon Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions