fix(csi): recover NodeStageVolume when staging was interrupted before the stash was written - #384
Open
boddumanohar wants to merge 2 commits into
Open
boddumanohar wants to merge 2 commits into
boddumanohar wants to merge 2 commits into
Conversation
This was referenced Jul 30, 2026
noctarius
force-pushed
the
main
branch
2 times, most recently
from
September 9, 2026 10:21
60dceb7 to
fbaabe4
Compare
… the stash was written If stageVolume() fails after FormatAndMount but before StashVolumeContext (e.g. tune2fs aborting on an unrecognised ext4 feature flag), the mount exists but no VolumeContext stash is ever written. On the next NodeStageVolume retry, isStaged() reports true and IsDead() reports false (the backing device is often still connected at this point), so the driver took the "already staged" short-circuit and returned success forever without ever staging or stashing the volume. restageVolume() also couldn't recover this case on its own since it requires a stash to already exist. Detect the missing stash directly: force-unmount and fall through to a fresh stage, which rebuilds the VolumeContext from the request and writes the stash on success. Rebased onto main's csi-driver restructure (#497): the fix now lives in internal/csi/node/stage.go instead of pkg/spdk/nodeserver.go, using lookupVolumeContext and ns.mounter.ForceUnmount/IsDead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
boddumanohar
force-pushed
the
fix/csi-nodestagevolume-orphaned-mount
branch
from
September 11, 2026 13:49
1e52d68 to
1f95011
Compare
… no stash A volume whose stash was never written (NodeStageVolume interrupted between FormatAndMount and StashVolumeContext, see the sibling fix in NodeStageVolume) had no way to ever be unstaged: NodeUnstageVolume hard-required the stash to rebuild an initiator and disconnect the NVMe-oF connection, so every retry hit the exact same "volume context JSON file not found" error, forever. That pins the volume's VolumeAttachment and permanently blocks its pod from being rescheduled to another node. There is no NQN or model left to safely identify and tear down the volume's NVMe-oF connection without the stash, so guessing at it risks disconnecting the wrong thing on a shared subsystem. Instead, proceed with the unstage and leave a potential connection leak for manual or guardian-driven cleanup: blocking a volume forever is worse than a leak that can be found and cleared later. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
If
NodeStageVolumefails after formatting/mounting a volume but before writing its context stash (e.g. a crash/restart mid-call, ortune2fsfailing on an unrecognised ext4 feature flag), the mount exists but the stash never gets written. On retry, the driver sees the mount, assumes "already staged", and returns success without ever staging or restaging — the volume is stuck broken. Downstream,NodeUnstageVolumethen also permanently fails on the same volume (volume context JSON file not found), since it hard-requires that stash to disconnect the NVMe-oF connection — blocking the pod from ever being rescheduled.Fix
NodeStageVolume: detect the missing stash. If a mount exists but has no stash, force-unmount and fall through to a fresh stage, which rebuilds the VolumeContext and writes the stash.NodeUnstageVolume: if the stash is missing, don't block forever. There's no NQN/model left to safely identify and disconnect the NVMe-oF connection, so proceed with the unstage anyway (leaving a possible connection leak for manual/guardian cleanup) rather than permanently pinning the volume.Testing
go build,go vet,golangci-lint,go testall passNodeUnstageVolumefixNodeStageVolumebug on a live 4-node k3s cluster (deleted the stash file under a healthy mount) and confirmed the fix re-stages instead of silently returning success🤖 Generated with Claude Code