operator: exempt stuck pods' claims from the PVCUnbinder pvc-rebinding gate

[RafalKorepta]

Problem

The PVCUnbinder's Gate 3 (pvc-rebinding) defers any unbind while any claim in the cluster has an empty spec.volumeName, assuming an in-progress rebind that will settle shortly. That assumption is false for claims owned by Pods that are themselves stuck, producing a three-way circular wait that deadlocks the unbinder:

1. Gate 3 defers because a stuck Pod's datadir claim has no volumeName,
2. that claim can never bind until its Pod schedules (WaitForFirstConsumer), and
3. the Pod can never schedule until the unbinder frees its mis-pinned sibling claim — the very action Gate 3 is deferring.

Production impact (cluster d99mur9p5l53r2240m3g, 2026-07-12): a fresh AWS cluster never bootstrapped. broker-1's shadow-index-cache PV was provisioned onto broker-0's node; the pod could never fit there (didn't have free ports) nor anywhere else (didn't match PersistentVolume's node affinity). The unbinder logged a PVC in this cluster has no volumeName yet; deferring every 30s for ~2h while brokers 0/2 looped in cluster_discovery waiting for the missing seed. Manual PVC deletion was required.

Fix

Exempt claims owned by Pods stuck Pending with the unbinder's scheduling-failure signature (schedulingFailureRE — the same weak form ShouldRemediate accepts), the reconciled Pod first among them. Same reasoning as Gate 0's own-claim exemption (inFlightClaimOwnedBy), which exists for exactly this deadlock class.

• Symmetric across victims: two simultaneously mis-pinned brokers don't mutually defer on each other's unbound claims (adversarial review of the initial own-claims-only version found and empirically reproduced that variant). Destructive work stays serialized by Gate 0.
• Deliberately weak signature: modern schedulers prefix every total scheduling failure with 0/N nodes are available: and don't reliably name volume affinity, so a mid-rebind Pod can also be exempted. Safe with rebinding off: unbinder-initiated rebinds still defer siblings through Gate 0's uncached settle check, the destructive pipeline only touches the reconciled Pod's own claims, and no Available PV is ever produced for a floating claim to mis-pair with.
• --allow-pv-rebinding keeps the old conservative behavior (no exemption): that mode floats freed PVs as live binding candidates (see FreedPVAnnotation / INC-2818), where acting while any claim is unbound could pair it with a disk it was never meant to hold.
• Orphaned claims (no Pod) keep deferring — conservative, alertable via the gate metric.

Also:

• The Gate 3 log line and Event now name the gating claim (attribution of the production incident took hours partly because the log never said which claim was unbound).
• Updated the stale PVCUnbinderGateDeferred metric doc comment.
• Fixed the withPVC test fixture to produce unique volume names (previously two calls produced duplicate name: data volumes — invalid against a real API server).

Testing

TDD; each new subtest was watched failing against the old code first:

• own unbound WFFC claim does not deadlock the unbind — the production scenario end-to-end through Reconcile (mis-pinned bound claim deleted, PV Retain + in-flight annotations, pod deleted, unbound claim untouched).
• two stuck pods with unbound WFFC claims do not mutually deadlock — the ≥2-victim variant.
• allow-pv-rebinding keeps the conservative deferral even for own claims.
• sibling stuck on a non-affinity total scheduling failure is also exempted (weak signature) — pins the deliberate breadth.
• sibling unbound claim still defers — pins serialization, and asserts via events.FakeRecorder that the pvc-rebinding gate specifically fired and the event names the gating claim.

go test ./internal/controller/pvcunbinder/ ./internal/observability/ green; gofmt/go vet clean.

The change was additionally put through a multi-agent adversarial review (10 finder angles → per-finding verification → sweep); all confirmed correctness findings are addressed here or listed below.

Known pre-existing limitations (out of scope, surfaced by the review)

• An orphaned never-bound claim (aborted scale-up, then scale-in — STS scale-in retains PVCs) still defers all unbinds in its cluster forever.
• The same mis-provisioning class deadlocks at Gate 2 (multi-node) instead when two victims' PVs land on two different occupied nodes: Gate 2 classifies it as a K8s-wide event and defers to a natural recovery that WFFC mis-provisioning doesn't have.

🤖 Generated with Claude Code

[secpanda]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[hidalgopl]

I'm having a hard time understanding what this function does:

1. We get all PVCs for given pod argument (regardless of their status / being stuck) (from StsPVCs(pod))
2. Then we list all the pods for given Redpanda cluster and we add all stuck PVCs for these pods?
   Is this correct?