From 8080bb2b8a1820a91c7d914ee82db9400a7cb379 Mon Sep 17 00:00:00 2001 From: Rafal Korepta Date: Sun, 12 Jul 2026 15:24:39 +0200 Subject: [PATCH] operator: exempt stuck pods' claims from the PVCUnbinder pvc-rebinding gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Gate 3 (pvc-rebinding) defers any unbind while a claim in the cluster is unbound, assuming an in-progress rebind that will settle shortly. That assumption is false for claims owned by Pods that are themselves stuck Pending on a scheduling failure, and deadlocks the unbinder in a three-way circular wait: 1. Gate 3 defers because a stuck Pod's datadir claim has no volumeName, 2. that claim can never bind until its Pod schedules (WaitForFirstConsumer), and 3. the Pod can never schedule until the unbinder frees its mis-pinned sibling claim (e.g. a shadow-index-cache PV provisioned onto an already-occupied node) — the very action Gate 3 is deferring. Hit in production by a fresh AWS cluster that never bootstrapped: broker-1's shadow-index-cache PV landed on broker-0's node, the pod could never fit there ("didn't have free ports") nor anywhere else ("didn't match PersistentVolume's node affinity"), and the unbinder logged "a PVC in this cluster has no volumeName yet; deferring" every 30s for hours while the remaining brokers looped in cluster discovery waiting for the missing seed. Exempt a claim from Gate 3's deferral only under a full proof chain, evaluated identically for the reconciled Pod and each sibling (so two simultaneously mis-pinned brokers don't mutually defer on each other's unbound claims, while destructive work stays serialized by Gate 0): - the Pod passes ShouldRemediate in full — Selector, StatefulSet ownership, Pending, the weak scheduling-failure signature (modern schedulers don't reliably name volume affinity in the message), and the unbind Timeout measured against the Unschedulable condition's LastTransitionTime; - the Pod holds a Bound claim on a HostPath/Local PV — bound is proven by the PV's ClaimRef back-reference (namespace, name, and UID all matching the claim), never by the claim's user-settable volumeName alone — whose NodeAffinity-eligible node set (every hostname the selector accepts, resolved by the kubernetes.io/hostname LABEL rather than the Node object name; unresolvable selector shapes fail closed) is entirely unavailable: node gone, cordoned, or NotReady/unreachable — with the Ready condition and BOTH effect twins of the not-ready/unreachable taints (the node lifecycle controller applies NoExecute and NoSchedule together) judged through one canonical NoExecute toleration lens, so the auto-injected TolerationSeconds grace tolerations don't count while tolerate-forever pods (--broker-pod-node-unavailable-toleration=-1s, whose contract is that only Node deletion signals permanent loss) never treat transient unreachability as proof — or occupied by a live pod matching one of the Pod's own RequiredDuringScheduling anti-affinity terms (only terms whose full semantics are interpretable: hostname topology, own-namespace scope including the v1 Cluster's explicit single-namespace shape, no matchLabelKeys — anything else fails closed; candidate occupants are every pod in the namespace, so a custom term that selects a different workload still proves its occupant); - only the Pod's own unbound claims that would bind under a WaitForFirstConsumer StorageClass are exempted, with the class resolved exactly as Kubernetes' GetPersistentVolumeClaimClass resolves it (beta annotation first, by key presence, then spec.storageClassName; no fallback to the cluster's current default). Every read that can grant an exemption goes through the uncached APIReader — the reconciled Pod itself (re-qualified via ShouldRemediate on an uncached re-read decoded into a fresh object — decoding into the cache-populated one would merge and keep stale fields the fresh response omits — before any evidence or destructive step, so a stale informer copy of a since-resolved Pod can neither supply exemption evidence nor reach the PVC deletes), node state, occupant pods, sibling discovery, the per-claim PVC evidence Gets, and the StorageClass binding-mode Get (the sole exception, the mis-pinned PV Get, only consults fields that never change on a live Bound PV) — so a lagging informer can neither fabricate a mis-pin nor keep an already-resolved sibling looking stuck. Deferral-only reads stay cached. The node/occupant/PVC/StorageClass evidence chain is computed lazily, only when some claim is actually unbound, so the common all-claims-bound remediation path adds only the one uncached Pod re-read (which runs on every qualifying reconcile, disable flag or not); evidence-read failures (e.g. a 403 on the nodes LIST under RBAC version skew) fall back fail-safe to the conservative deferral with the gate metric and Event instead of failing the reconcile (context cancellation still surfaces as an error); and even with every unbound claim exempted, the reconciled Pod must additionally hold its own mis-pin proof before deletion proceeds — a sibling's deadlock never transitively authorizes destroying a Pod that isn't part of one. Exemption-based gate passes leave the same paper trail deferrals do (the pvc_unbinder_gate_exempted_total metric, a Warning PVCUnbinderGateExempted Event naming the exempted claims — capped by name count and total rendered length so the note never exceeds the events.k8s.io/v1 1024-char limit, past which the broadcaster silently drops the Event — and a log line carrying the full list), recorded only when the reconcile actually proceeds — a reconcile still deferred by the durable freed-pv gate does not count gate passes every 30s — deferral messages name a deterministic (sorted-first) gating claim, siblings owning none of the unbound claims are skipped without an evidence run, and claims with no Pod at all keep deferring. A --disable-pvc-rebinding-gate-exemption flag (operator run/multicluster and sidecar binaries; exposed in the redpanda chart as statefulset.sideCars.pvcUnbinder.disableStuckClaimExemption) turns just the exemption off as an escape hatch, keeping the rest of the unbinder running. Under the deprecated --allow-pv-rebinding flag no exemption applies: that mode floats freed PVs as live binding candidates (see FreedPVAnnotation / INC-2818), where acting while any claim is unbound could pair it with a disk it was never meant to hold, so the original conservative deferral is kept there. The freed-pv gate resolves a freed PV's pinned node by the kubernetes.io/hostname label too — a name-based Get would report a live node as gone under kubelet --hostname-override and silently disengage the gate. Also name the gating claim in the Gate 3 log line and Event (the production incident took hours to attribute partly because the log never said WHICH claim was unbound), update the stale PVCUnbinderGateDeferred metric doc, grant the unbinder storageclasses get and nodes list, fix the withPVC test fixture to produce unique volume names, and rewrite pvcunbinder.go's pre-existing doc comments file-wide into substantially shorter plain-English sentences (same invariants). Known limitations, out of scope here: an orphaned never-bound claim (aborted scale-up, then scale-in) still defers all unbinds in its cluster; deployments whose pod-anti-affinity yields no interpretable required hostname-topology term (type soft, or hard/custom terms of any other shape — terms are judged by shape, not by which chart option produced them) get no occupancy-based exemption (Gate 3 keeps deferring, alertable via the gate metric); and the same mis-provisioning class deadlocks at Gate 2 instead when two victims' PVs land on two different occupied nodes. Co-Authored-By: Claude Opus 4.8 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01XG2iDDtCFzz4QSo3FMsAyR --- ...nda-Changed-20260712-pvcunbinder-rbac.yaml | 4 + ...-20260712-pvcunbinder-gate3-own-claim.yaml | 57 + charts/redpanda/chart/README.md | 8 + .../chart/files/pvcunbinder.ClusterRole.yaml | 7 + charts/redpanda/chart/templates/_chart.go.tpl | 2 +- .../chart/templates/_configmap.go.tpl | 4 +- .../redpanda/chart/templates/_secrets.go.tpl | 4 +- .../templates/_service.loadbalancer.go.tpl | 2 +- .../chart/templates/_statefulset.go.tpl | 11 +- .../redpanda/chart/templates/_tlsroute.go.tpl | 2 +- .../redpanda/chart/templates/_values.go.tpl | 60 +- charts/redpanda/chart/values.schema.json | 3 + charts/redpanda/chart/values.yaml | 7 + charts/redpanda/statefulset.go | 4 + charts/redpanda/testdata/template-cases.txtar | 9 + charts/redpanda/values.go | 5 + charts/redpanda/values_partial.gen.go | 5 +- .../files/rbac/pvcunbinder.ClusterRole.yaml | 7 + .../testdata/template-cases.golden.txtar | 1225 ++++++++++ operator/cmd/multicluster/multicluster.go | 11 +- operator/cmd/run/run.go | 11 +- operator/cmd/sidecar/sidecar.go | 11 +- operator/config/rbac/bases/operator/role.yaml | 6 + .../config/rbac/itemized/pvcunbinder.yaml | 7 + .../controller/pvcunbinder/pvcunbinder.go | 1346 ++++++++--- .../pvcunbinder/pvcunbinder_internal_test.go | 1974 ++++++++++++++++- operator/internal/observability/metrics.go | 33 +- 27 files changed, 4429 insertions(+), 396 deletions(-) create mode 100644 .changes/unreleased/charts-redpanda-Changed-20260712-pvcunbinder-rbac.yaml create mode 100644 .changes/unreleased/operator-Fixed-20260712-pvcunbinder-gate3-own-claim.yaml diff --git a/.changes/unreleased/charts-redpanda-Changed-20260712-pvcunbinder-rbac.yaml b/.changes/unreleased/charts-redpanda-Changed-20260712-pvcunbinder-rbac.yaml new file mode 100644 index 000000000..292f99a2c --- /dev/null +++ b/.changes/unreleased/charts-redpanda-Changed-20260712-pvcunbinder-rbac.yaml @@ -0,0 +1,4 @@ +project: charts/redpanda +kind: Changed +body: The PVCUnbinder sidecar's ClusterRole (rendered when both `rbac.enabled` and `statefulset.sideCars.pvcUnbinder.enabled` are set; with `rbac.enabled: false` the new verbs must be added to your out-of-band role, or the unbinder falls back to its conservative deferral) additionally grants cluster-scoped `nodes list` and `storageclasses get`. Both are read-only and consumed by the unbinder's new pvc-rebinding-gate exemption, which resolves a PV's pinned nodes by their `kubernetes.io/hostname` label and checks the unbound claim's StorageClass binding mode before acting. A new `statefulset.sideCars.pvcUnbinder.disableStuckClaimExemption` value renders the sidecar's `--disable-pvc-rebinding-gate-exemption` escape hatch, turning just that exemption off while keeping the PVCUnbinder running. +time: 2026-07-12T15:23:55.000000+02:00 diff --git a/.changes/unreleased/operator-Fixed-20260712-pvcunbinder-gate3-own-claim.yaml b/.changes/unreleased/operator-Fixed-20260712-pvcunbinder-gate3-own-claim.yaml new file mode 100644 index 000000000..b4e4fe6f6 --- /dev/null +++ b/.changes/unreleased/operator-Fixed-20260712-pvcunbinder-gate3-own-claim.yaml @@ -0,0 +1,57 @@ +project: operator +kind: Fixed +body: |- + The PVCUnbinder's pvc-rebinding gate (Gate 3) no longer defers on + unbound claims owned by Pods that are provably deadlocked on the + unbinder's own remediation. Treating those claims as in-progress + rebinds deadlocked clusters that use WaitForFirstConsumer binding — + a stuck Pod's unbound claim can never bind until the Pod schedules, + the Pod can never schedule until its mis-pinned sibling claim is + unbound, and the gate deferred exactly that unbind forever. A fresh + cluster whose broker's shadow-index-cache PV was provisioned onto + an already-occupied node never bootstrapped, with the unbinder + logging "a PVC in this cluster has no volumeName yet; deferring" + every 30s; the exemption is symmetric so multiple simultaneously + mis-pinned brokers cannot mutually deadlock either. The exemption + applies only under a strict proof chain: the Pod is stuck Pending + past the unbind timeout, holds a Bound claim on a HostPath/Local PV + whose every eligible node is provably unavailable (gone, cordoned, + NotReady/unreachable net of unconditional tolerations — pods under + --broker-pod-node-unavailable-toleration=-1s never treat transient + unreachability as loss — or occupied by a pod matched by the stuck + Pod's own hard hostname-scoped anti-affinity), the unbound claim + resolves to a WaitForFirstConsumer StorageClass, and the reconciled + Pod additionally holds its own mis-pin proof before any deletion + proceeds. The occupancy-based proof requires a hard (Required) + anti-affinity term scoped to the kubernetes.io/hostname topology + in the pod's own namespace with no matchLabelKeys — terms are + judged by that shape whether statefulset.podAntiAffinity type + "hard" or "custom" produced them, while type "soft" yields no + required terms and never qualifies (the other proof legs — node + gone, cordoned, or unreachable — still apply); clusters deadlocked + purely on an occupied node without such a term keep deferring, + visible via the pvc-rebinding gate metric and Event. The + two-victims-on-two-nodes variant of the same + mis-provisioning is a known remaining gap: it defers at the + multi-node gate instead and still requires manual PVC deletion. + Orphaned claims with no Pod keep deferring the unbind, + unbinder-initiated rebinds stay serialized by the in-flight gate's + uncached settle check, the deferral log/Event now names the gating + claim, every exemption-based gate pass that proceeds to remediation + is recorded by a Warning PVCUnbinderGateExempted Event, a log + line, and the new pvc_unbinder_gate_exempted_total metric (a pass + still held by the freed-pv gate records that gate's deferral + instead), and under the deprecated + --allow-pv-rebinding flag the original conservative behavior (no + exemption at all) is kept because freed Available PVs make acting + while any claim is unbound unsafe. A new + --disable-pvc-rebinding-gate-exemption flag (operator and sidecar; + in the redpanda chart set + `statefulset.sideCars.pvcUnbinder.disableStuckClaimExemption`) + turns just the exemption off as an escape hatch while keeping the + rest of the unbinder running. The unbinder's ClusterRole + additionally grants nodes list and storageclasses get for the + exemption's evidence reads; if RBAC is managed out-of-band and lags + the upgrade, evidence reads fall back to the conservative deferral + (fail-safe) rather than erroring. +time: 2026-07-12T15:23:50.000000+02:00 diff --git a/charts/redpanda/chart/README.md b/charts/redpanda/chart/README.md index e9139a24b..cc35b122c 100644 --- a/charts/redpanda/chart/README.md +++ b/charts/redpanda/chart/README.md @@ -893,8 +893,16 @@ DEPRECATED: Please use statefulset.sideCars.brokerDecommissioner and statefulset **Default:** `"v26.2.1-beta.3"` +### [statefulset.sideCars.pvcUnbinder.disableStuckClaimExemption](https://artifacthub.io/packages/helm/redpanda-data/redpanda?modal=values&path=statefulset.sideCars.pvcUnbinder.disableStuckClaimExemption) + +Renders `--disable-pvc-rebinding-gate-exemption`: turns off the pvc-rebinding gate's stuck-claim exemption (an escape hatch if its proof chain misfires in your environment) while keeping the rest of the PVCUnbinder running. + +**Default:** `false` + ### [statefulset.sideCars.pvcUnbinder.enabled](https://artifacthub.io/packages/helm/redpanda-data/redpanda?modal=values&path=statefulset.sideCars.pvcUnbinder.enabled) +Enables the PVCUnbinder sidecar controller. Note: with `rbac.enabled`, this renders a ClusterRole that grants the Pod's ServiceAccount cluster-wide read access to Nodes, regardless of the other pvcUnbinder settings. + **Default:** `false` ### [statefulset.sideCars.pvcUnbinder.unbindAfter](https://artifacthub.io/packages/helm/redpanda-data/redpanda?modal=values&path=statefulset.sideCars.pvcUnbinder.unbindAfter) diff --git a/charts/redpanda/chart/files/pvcunbinder.ClusterRole.yaml b/charts/redpanda/chart/files/pvcunbinder.ClusterRole.yaml index a969b3fbf..a70f0fdd1 100644 --- a/charts/redpanda/chart/files/pvcunbinder.ClusterRole.yaml +++ b/charts/redpanda/chart/files/pvcunbinder.ClusterRole.yaml @@ -10,6 +10,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -50,3 +51,9 @@ rules: - get - list - watch + - apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get diff --git a/charts/redpanda/chart/templates/_chart.go.tpl b/charts/redpanda/chart/templates/_chart.go.tpl index d94f91f8f..081e2019d 100644 --- a/charts/redpanda/chart/templates/_chart.go.tpl +++ b/charts/redpanda/chart/templates/_chart.go.tpl @@ -5,7 +5,7 @@ {{- $dot := (index .a 0) -}} {{- range $_ := (list 1) -}} {{- $_is_returning := false -}} -{{- $state := (mustMergeOverwrite (dict "Release" (coalesce nil) "Files" (coalesce nil) "Chart" (coalesce nil) "Values" (dict "nameOverride" "" "fullnameOverride" "" "clusterDomain" "" "commonLabels" (coalesce nil) "image" (dict "repository" "" "tag" "") "service" (coalesce nil) "license_key" "" "auditLogging" (dict "enabled" false "listener" "" "partitions" 0 "enabledEventTypes" (coalesce nil) "excludedTopics" (coalesce nil) "excludedPrincipals" (coalesce nil) "clientMaxBufferSize" 0 "queueDrainIntervalMs" 0 "queueMaxBufferSizePerShard" 0 "replicationFactor" 0) "enterprise" (dict "license" "") "rackAwareness" (dict "enabled" false "nodeAnnotation" "") "console" (dict) "auth" (dict "sasl" (coalesce nil)) "tls" (dict "enabled" false "certs" (coalesce nil)) "external" (dict "addresses" (coalesce nil) "annotations" (coalesce nil) "domain" (coalesce nil) "enabled" false "type" "" "prefixTemplate" "" "sourceRanges" (coalesce nil) "service" (dict "enabled" false) "externalDns" (coalesce nil)) "logging" (dict "logLevel" "" "usageStats" (dict "enabled" false "clusterId" (coalesce nil))) "monitoring" (dict "enabled" false "scrapeInterval" "" "labels" (coalesce nil) "tlsConfig" (coalesce nil) "enableHttp2" (coalesce nil)) "resources" (dict "cpu" (dict "cores" "0" "overprovisioned" (coalesce nil)) "memory" (dict "enable_memory_locking" (coalesce nil) "container" (dict "min" (coalesce nil) "max" "0") "redpanda" (coalesce nil))) "storage" (dict "hostPath" "" "tiered" (dict "credentialsSecretRef" (dict "accessKey" (coalesce nil) "secretKey" (coalesce nil)) "config" (coalesce nil) "hostPath" "" "mountType" "" "persistentVolume" (dict "annotations" (coalesce nil) "enabled" false "labels" (coalesce nil) "nameOverwrite" "" "size" "" "storageClass" "")) "persistentVolume" (coalesce nil) "tieredConfig" (coalesce nil) "tieredStorageHostPath" "" "tieredStoragePersistentVolume" (coalesce nil)) "post_install_job" (dict "enabled" false "labels" (coalesce nil) "annotations" (coalesce nil) "podTemplate" (dict)) "statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "") "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "serviceAccount" (dict "annotations" (coalesce nil) "create" false "name" "") "rbac" (dict "enabled" false "rpkDebugBundle" false "annotations" (coalesce nil)) "tuning" (dict) "listeners" (dict "admin" (dict "enabled" false "external" (coalesce nil) "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil))) "http" (dict "enabled" false "external" (coalesce nil) "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil))) "kafka" (dict "enabled" false "external" (coalesce nil) "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil))) "schemaRegistry" (dict "enabled" false "external" (coalesce nil) "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil))) "rpc" (dict "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil)))) "config" (dict "cluster" (coalesce nil) "extraClusterConfiguration" (coalesce nil) "node" (coalesce nil) "rpk" (coalesce nil) "schema_registry_client" (coalesce nil) "pandaproxy_client" (coalesce nil) "tunable" (coalesce nil)) "tests" (coalesce nil) "force" false "podTemplate" (dict)) "BootstrapUserSecret" (coalesce nil) "BootstrapUserPassword" "" "StatefulSetPodLabels" (coalesce nil) "StatefulSetSelector" (coalesce nil) "Pools" (coalesce nil) "Dot" (coalesce nil) "ViaOperator" false "CloudEnvironment" "" "OperatorVersion" "") (dict "Release" $dot.Release "Files" $dot.Files "Chart" $dot.Chart "Values" $dot.Values.AsMap "Dot" $dot)) -}} +{{- $state := (mustMergeOverwrite (dict "Release" (coalesce nil) "Files" (coalesce nil) "Chart" (coalesce nil) "Values" (dict "nameOverride" "" "fullnameOverride" "" "clusterDomain" "" "commonLabels" (coalesce nil) "image" (dict "repository" "" "tag" "") "service" (coalesce nil) "license_key" "" "auditLogging" (dict "enabled" false "listener" "" "partitions" 0 "enabledEventTypes" (coalesce nil) "excludedTopics" (coalesce nil) "excludedPrincipals" (coalesce nil) "clientMaxBufferSize" 0 "queueDrainIntervalMs" 0 "queueMaxBufferSizePerShard" 0 "replicationFactor" 0) "enterprise" (dict "license" "") "rackAwareness" (dict "enabled" false "nodeAnnotation" "") "console" (dict) "auth" (dict "sasl" (coalesce nil)) "tls" (dict "enabled" false "certs" (coalesce nil)) "external" (dict "addresses" (coalesce nil) "annotations" (coalesce nil) "domain" (coalesce nil) "enabled" false "type" "" "prefixTemplate" "" "sourceRanges" (coalesce nil) "service" (dict "enabled" false) "externalDns" (coalesce nil)) "logging" (dict "logLevel" "" "usageStats" (dict "enabled" false "clusterId" (coalesce nil))) "monitoring" (dict "enabled" false "scrapeInterval" "" "labels" (coalesce nil) "tlsConfig" (coalesce nil) "enableHttp2" (coalesce nil)) "resources" (dict "cpu" (dict "cores" "0" "overprovisioned" (coalesce nil)) "memory" (dict "enable_memory_locking" (coalesce nil) "container" (dict "min" (coalesce nil) "max" "0") "redpanda" (coalesce nil))) "storage" (dict "hostPath" "" "tiered" (dict "credentialsSecretRef" (dict "accessKey" (coalesce nil) "secretKey" (coalesce nil)) "config" (coalesce nil) "hostPath" "" "mountType" "" "persistentVolume" (dict "annotations" (coalesce nil) "enabled" false "labels" (coalesce nil) "nameOverwrite" "" "size" "" "storageClass" "")) "persistentVolume" (coalesce nil) "tieredConfig" (coalesce nil) "tieredStorageHostPath" "" "tieredStoragePersistentVolume" (coalesce nil)) "post_install_job" (dict "enabled" false "labels" (coalesce nil) "annotations" (coalesce nil) "podTemplate" (dict)) "statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "" "disableStuckClaimExemption" false) "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "serviceAccount" (dict "annotations" (coalesce nil) "create" false "name" "") "rbac" (dict "enabled" false "rpkDebugBundle" false "annotations" (coalesce nil)) "tuning" (dict) "listeners" (dict "admin" (dict "enabled" false "external" (coalesce nil) "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil))) "http" (dict "enabled" false "external" (coalesce nil) "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil))) "kafka" (dict "enabled" false "external" (coalesce nil) "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil))) "schemaRegistry" (dict "enabled" false "external" (coalesce nil) "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil))) "rpc" (dict "port" 0 "tls" (dict "enabled" (coalesce nil) "cert" "" "requireClientAuth" false "trustStore" (coalesce nil)))) "config" (dict "cluster" (coalesce nil) "extraClusterConfiguration" (coalesce nil) "node" (coalesce nil) "rpk" (coalesce nil) "schema_registry_client" (coalesce nil) "pandaproxy_client" (coalesce nil) "tunable" (coalesce nil)) "tests" (coalesce nil) "force" false "podTemplate" (dict)) "BootstrapUserSecret" (coalesce nil) "BootstrapUserPassword" "" "StatefulSetPodLabels" (coalesce nil) "StatefulSetSelector" (coalesce nil) "Pools" (coalesce nil) "Dot" (coalesce nil) "ViaOperator" false "CloudEnvironment" "" "OperatorVersion" "") (dict "Release" $dot.Release "Files" $dot.Files "Chart" $dot.Chart "Values" $dot.Values.AsMap "Dot" $dot)) -}} {{- $_ := (get (fromJson (include "redpanda.RenderState.FetchBootstrapUser" (dict "a" (list $state)))) "r") -}} {{- $_ := (get (fromJson (include "redpanda.RenderState.FetchStatefulSetPodSelector" (dict "a" (list $state)))) "r") -}} {{- $manifests := (get (fromJson (include "redpanda.renderResources" (dict "a" (list $state)))) "r") -}} diff --git a/charts/redpanda/chart/templates/_configmap.go.tpl b/charts/redpanda/chart/templates/_configmap.go.tpl index 3d56f5b3a..146d3c25e 100644 --- a/charts/redpanda/chart/templates/_configmap.go.tpl +++ b/charts/redpanda/chart/templates/_configmap.go.tpl @@ -5,7 +5,7 @@ {{- $state := (index .a 0) -}} {{- range $_ := (list 1) -}} {{- $_is_returning := false -}} -{{- $cms := (list (get (fromJson (include "redpanda.RedpandaConfigMap" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "") "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r")) -}} +{{- $cms := (list (get (fromJson (include "redpanda.RedpandaConfigMap" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "" "disableStuckClaimExemption" false) "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r")) -}} {{- range $_, $set := $state.Pools -}} {{- $cms = (concat (default (list) $cms) (list (get (fromJson (include "redpanda.RedpandaConfigMap" (dict "a" (list $state $set)))) "r"))) -}} {{- end -}} @@ -340,7 +340,7 @@ {{- $port := (index .a 1) -}} {{- range $_ := (list 1) -}} {{- $_is_returning := false -}} -{{- $bl := (get (fromJson (include "redpanda.brokersFor" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "") "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)) $port)))) "r") -}} +{{- $bl := (get (fromJson (include "redpanda.brokersFor" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "" "disableStuckClaimExemption" false) "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)) $port)))) "r") -}} {{- range $_, $set := $state.Pools -}} {{- $bl = (concat (default (list) $bl) (default (list) (get (fromJson (include "redpanda.brokersFor" (dict "a" (list $state $set $port)))) "r"))) -}} {{- end -}} diff --git a/charts/redpanda/chart/templates/_secrets.go.tpl b/charts/redpanda/chart/templates/_secrets.go.tpl index 29a9cedcc..76f980d23 100644 --- a/charts/redpanda/chart/templates/_secrets.go.tpl +++ b/charts/redpanda/chart/templates/_secrets.go.tpl @@ -11,8 +11,8 @@ {{- if (ne (toJson $saslUsers_1) "null") -}} {{- $secrets = (concat (default (list) $secrets) (list $saslUsers_1)) -}} {{- end -}} -{{- $secrets = (concat (default (list) $secrets) (list (get (fromJson (include "redpanda.SecretConfigurator" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "") "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)) (0 | int))))) "r"))) -}} -{{- $fsValidator_2 := (get (fromJson (include "redpanda.SecretFSValidator" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "") "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r") -}} +{{- $secrets = (concat (default (list) $secrets) (list (get (fromJson (include "redpanda.SecretConfigurator" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "" "disableStuckClaimExemption" false) "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)) (0 | int))))) "r"))) -}} +{{- $fsValidator_2 := (get (fromJson (include "redpanda.SecretFSValidator" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "" "disableStuckClaimExemption" false) "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r") -}} {{- if (ne (toJson $fsValidator_2) "null") -}} {{- $secrets = (concat (default (list) $secrets) (list $fsValidator_2)) -}} {{- end -}} diff --git a/charts/redpanda/chart/templates/_service.loadbalancer.go.tpl b/charts/redpanda/chart/templates/_service.loadbalancer.go.tpl index 97ec3471f..59174c358 100644 --- a/charts/redpanda/chart/templates/_service.loadbalancer.go.tpl +++ b/charts/redpanda/chart/templates/_service.loadbalancer.go.tpl @@ -30,7 +30,7 @@ {{- break -}} {{- end -}} {{- $services := (coalesce nil) -}} -{{- $pods := (get (fromJson (include "redpanda.PodNames" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "") "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r") -}} +{{- $pods := (get (fromJson (include "redpanda.PodNames" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "" "disableStuckClaimExemption" false) "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r") -}} {{- range $_, $set := $state.Pools -}} {{- $pods = (concat (default (list) $pods) (default (list) (get (fromJson (include "redpanda.PodNames" (dict "a" (list $state $set)))) "r"))) -}} {{- end -}} diff --git a/charts/redpanda/chart/templates/_statefulset.go.tpl b/charts/redpanda/chart/templates/_statefulset.go.tpl index 524d3de4a..dd61ea954 100644 --- a/charts/redpanda/chart/templates/_statefulset.go.tpl +++ b/charts/redpanda/chart/templates/_statefulset.go.tpl @@ -582,12 +582,15 @@ chroot /host /bin/bash -c ' {{- end -}} {{- if $pool.Statefulset.sideCars.pvcUnbinder.enabled -}} {{- $args = (concat (default (list) $args) (default (list) (list `--run-pvc-unbinder` (printf "--pvc-unbinder-timeout=%s" $pool.Statefulset.sideCars.pvcUnbinder.unbindAfter)))) -}} +{{- if $pool.Statefulset.sideCars.pvcUnbinder.disableStuckClaimExemption -}} +{{- $args = (concat (default (list) $args) (list `--disable-pvc-rebinding-gate-exemption`)) -}} +{{- end -}} {{- end -}} {{- $args = (concat (default (list) $args) (default (list) $pool.Statefulset.sideCars.args)) -}} {{- $volumeMounts := (concat (default (list) (get (fromJson (include "redpanda.CommonMounts" (dict "a" (list $state)))) "r")) (list (mustMergeOverwrite (dict "name" "" "mountPath" "") (dict "name" "config" "mountPath" "/etc/redpanda")) (mustMergeOverwrite (dict "name" "" "mountPath" "") (dict "name" "base-config" "mountPath" "/tmp/base-config" "readOnly" true)) (mustMergeOverwrite (dict "name" "" "mountPath" "") (dict "name" "kube-api-access" "mountPath" "/var/run/secrets/kubernetes.io/serviceaccount" "readOnly" true)))) -}} -{{- $_1232_uid_gid := (get (fromJson (include "redpanda.securityContextUidGid" (dict "a" (list $state $pool "sidecar")))) "r") -}} -{{- $uid := ((index $_1232_uid_gid 0) | int64) -}} -{{- $gid := ((index $_1232_uid_gid 1) | int64) -}} +{{- $_1236_uid_gid := (get (fromJson (include "redpanda.securityContextUidGid" (dict "a" (list $state $pool "sidecar")))) "r") -}} +{{- $uid := ((index $_1236_uid_gid 0) | int64) -}} +{{- $gid := ((index $_1236_uid_gid 1) | int64) -}} {{- $_is_returning = true -}} {{- (dict "r" (mustMergeOverwrite (dict "name" "" "resources" (dict)) (dict "name" "sidecar" "image" (printf `%s:%s` $pool.Statefulset.sideCars.image.repository $pool.Statefulset.sideCars.image.tag) "command" (list `/redpanda-operator`) "args" (concat (default (list) (list `supervisor` `--`)) (default (list) $args)) "env" (concat (default (list) (get (fromJson (include "redpanda.rpkEnvVars" (dict "a" (list $state (coalesce nil))))) "r")) (default (list) (get (fromJson (include "redpanda.statefulSetRedpandaEnv" (dict "a" (list)))) "r"))) "volumeMounts" $volumeMounts "securityContext" (mustMergeOverwrite (dict) (dict "runAsUser" $uid "runAsGroup" $gid "runAsNonRoot" true "allowPrivilegeEscalation" false)) "readinessProbe" (mustMergeOverwrite (dict) (mustMergeOverwrite (dict) (dict "httpGet" (mustMergeOverwrite (dict "port" 0) (dict "path" "/healthz" "port" (8093 | int))))) (dict "failureThreshold" (3 | int) "initialDelaySeconds" (1 | int) "periodSeconds" (10 | int) "successThreshold" (1 | int) "timeoutSeconds" (0 | int)))))) | toJson -}} {{- break -}} @@ -630,7 +633,7 @@ chroot /host /bin/bash -c ' {{- $state := (index .a 0) -}} {{- range $_ := (list 1) -}} {{- $_is_returning := false -}} -{{- $sets := (list (get (fromJson (include "redpanda.StatefulSet" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "") "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r")) -}} +{{- $sets := (list (get (fromJson (include "redpanda.StatefulSet" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "" "disableStuckClaimExemption" false) "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r")) -}} {{- range $_, $set := $state.Pools -}} {{- $sets = (concat (default (list) $sets) (list (get (fromJson (include "redpanda.StatefulSet" (dict "a" (list $state $set)))) "r"))) -}} {{- end -}} diff --git a/charts/redpanda/chart/templates/_tlsroute.go.tpl b/charts/redpanda/chart/templates/_tlsroute.go.tpl index 48e6a5e02..f0122abe7 100644 --- a/charts/redpanda/chart/templates/_tlsroute.go.tpl +++ b/charts/redpanda/chart/templates/_tlsroute.go.tpl @@ -134,7 +134,7 @@ {{- $state := (index .a 0) -}} {{- range $_ := (list 1) -}} {{- $_is_returning := false -}} -{{- $pods := (get (fromJson (include "redpanda.PodNames" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "") "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r") -}} +{{- $pods := (get (fromJson (include "redpanda.PodNames" (dict "a" (list $state (mustMergeOverwrite (dict "Name" "" "Generation" "" "Statefulset" (dict "additionalSelectorLabels" (coalesce nil) "replicas" 0 "updateStrategy" (dict) "additionalRedpandaCmdFlags" (coalesce nil) "podTemplate" (dict) "budget" (dict "maxUnavailable" 0) "podAntiAffinity" (dict "topologyKey" "" "type" "" "weight" 0 "custom" (coalesce nil)) "sideCars" (dict "image" (dict "repository" "" "tag" "") "args" (coalesce nil) "pvcUnbinder" (dict "enabled" false "unbindAfter" "" "disableStuckClaimExemption" false) "brokerDecommissioner" (dict "enabled" false "decommissionAfter" "" "decommissionRequeueTimeout" "") "configWatcher" (dict "enabled" false) "rpkProfileWatcher" (dict "enabled" false) "controllers" (dict "image" (coalesce nil) "enabled" false "createRBAC" false "healthProbeAddress" "" "metricsAddress" "" "pprofAddress" "" "run" (coalesce nil))) "initContainers" (dict "fsValidator" (dict "enabled" false "expectedFS" "") "setDataDirOwnership" (dict "enabled" false) "configurator" (dict)) "initContainerImage" (dict "repository" "" "tag" "")) "ServiceAnnotations" (coalesce nil)) (dict "Statefulset" $state.Values.statefulset)))))) "r") -}} {{- range $_, $set := $state.Pools -}} {{- $pods = (concat (default (list) $pods) (default (list) (get (fromJson (include "redpanda.PodNames" (dict "a" (list $state $set)))) "r"))) -}} {{- end -}} diff --git a/charts/redpanda/chart/templates/_values.go.tpl b/charts/redpanda/chart/templates/_values.go.tpl index cde6e0f37..dae1779a6 100644 --- a/charts/redpanda/chart/templates/_values.go.tpl +++ b/charts/redpanda/chart/templates/_values.go.tpl @@ -684,9 +684,9 @@ {{- $seen := (dict) -}} {{- $deduped := (coalesce nil) -}} {{- range $_, $item := $items -}} -{{- $_1220___ok_11 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $seen $item.key false)))) "r") -}} -{{- $_ := (index $_1220___ok_11 0) -}} -{{- $ok_11 := (index $_1220___ok_11 1) -}} +{{- $_1225___ok_11 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $seen $item.key false)))) "r") -}} +{{- $_ := (index $_1225___ok_11 0) -}} +{{- $ok_11 := (index $_1225___ok_11 1) -}} {{- if $ok_11 -}} {{- continue -}} {{- end -}} @@ -909,9 +909,9 @@ {{- $name := (index .a 1) -}} {{- range $_ := (list 1) -}} {{- $_is_returning := false -}} -{{- $_1508_cert_ok := (get (fromJson (include "_shims.dicttest" (dict "a" (list $m $name (dict "enabled" (coalesce nil) "caEnabled" false "applyInternalDNSNames" (coalesce nil) "duration" "" "issuerRef" (coalesce nil) "secretRef" (coalesce nil) "clientSecretRef" (coalesce nil)))))) "r") -}} -{{- $cert := (index $_1508_cert_ok 0) -}} -{{- $ok := (index $_1508_cert_ok 1) -}} +{{- $_1513_cert_ok := (get (fromJson (include "_shims.dicttest" (dict "a" (list $m $name (dict "enabled" (coalesce nil) "caEnabled" false "applyInternalDNSNames" (coalesce nil) "duration" "" "issuerRef" (coalesce nil) "secretRef" (coalesce nil) "clientSecretRef" (coalesce nil)))))) "r") -}} +{{- $cert := (index $_1513_cert_ok 0) -}} +{{- $ok := (index $_1513_cert_ok 1) -}} {{- if (not $ok) -}} {{- $_ := (fail (printf "Certificate %q referenced, but not found in the tls.certs map" $name)) -}} {{- end -}} @@ -1403,9 +1403,9 @@ {{- $result := (dict) -}} {{- range $k, $v := $c -}} {{- if (not (empty $v)) -}} -{{- $_2079___ok_15 := (get (fromJson (include "_shims.asnumeric" (dict "a" (list $v)))) "r") -}} -{{- $_ := ((index $_2079___ok_15 0) | float64) -}} -{{- $ok_15 := (index $_2079___ok_15 1) -}} +{{- $_2084___ok_15 := (get (fromJson (include "_shims.asnumeric" (dict "a" (list $v)))) "r") -}} +{{- $_ := ((index $_2084___ok_15 0) | float64) -}} +{{- $ok_15 := (index $_2084___ok_15 1) -}} {{- if $ok_15 -}} {{- $_ := (set $result $k $v) -}} {{- else -}}{{- if (kindIs "bool" $v) -}} @@ -1431,9 +1431,9 @@ {{- $_is_returning := false -}} {{- $result := (dict) -}} {{- range $k, $v := $c -}} -{{- $_2099_b_16_ok_17 := (get (fromJson (include "_shims.typetest" (dict "a" (list "bool" $v false)))) "r") -}} -{{- $b_16 := (index $_2099_b_16_ok_17 0) -}} -{{- $ok_17 := (index $_2099_b_16_ok_17 1) -}} +{{- $_2104_b_16_ok_17 := (get (fromJson (include "_shims.typetest" (dict "a" (list "bool" $v false)))) "r") -}} +{{- $b_16 := (index $_2104_b_16_ok_17 0) -}} +{{- $ok_17 := (index $_2104_b_16_ok_17 1) -}} {{- if $ok_17 -}} {{- $_ := (set $result $k $b_16) -}} {{- continue -}} @@ -1476,15 +1476,15 @@ {{- $config := (index .a 1) -}} {{- range $_ := (list 1) -}} {{- $_is_returning := false -}} -{{- $_2144___hasAccessKey := (get (fromJson (include "_shims.dicttest" (dict "a" (list $config "cloud_storage_access_key" (coalesce nil))))) "r") -}} -{{- $_ := (index $_2144___hasAccessKey 0) -}} -{{- $hasAccessKey := (index $_2144___hasAccessKey 1) -}} -{{- $_2145___hasSecretKey := (get (fromJson (include "_shims.dicttest" (dict "a" (list $config "cloud_storage_secret_key" (coalesce nil))))) "r") -}} -{{- $_ := (index $_2145___hasSecretKey 0) -}} -{{- $hasSecretKey := (index $_2145___hasSecretKey 1) -}} -{{- $_2146___hasSharedKey := (get (fromJson (include "_shims.dicttest" (dict "a" (list $config "cloud_storage_azure_shared_key" (coalesce nil))))) "r") -}} -{{- $_ := (index $_2146___hasSharedKey 0) -}} -{{- $hasSharedKey := (index $_2146___hasSharedKey 1) -}} +{{- $_2149___hasAccessKey := (get (fromJson (include "_shims.dicttest" (dict "a" (list $config "cloud_storage_access_key" (coalesce nil))))) "r") -}} +{{- $_ := (index $_2149___hasAccessKey 0) -}} +{{- $hasAccessKey := (index $_2149___hasAccessKey 1) -}} +{{- $_2150___hasSecretKey := (get (fromJson (include "_shims.dicttest" (dict "a" (list $config "cloud_storage_secret_key" (coalesce nil))))) "r") -}} +{{- $_ := (index $_2150___hasSecretKey 0) -}} +{{- $hasSecretKey := (index $_2150___hasSecretKey 1) -}} +{{- $_2151___hasSharedKey := (get (fromJson (include "_shims.dicttest" (dict "a" (list $config "cloud_storage_azure_shared_key" (coalesce nil))))) "r") -}} +{{- $_ := (index $_2151___hasSharedKey 0) -}} +{{- $hasSharedKey := (index $_2151___hasSharedKey 1) -}} {{- $envvars := (coalesce nil) -}} {{- if (and (not $hasAccessKey) (get (fromJson (include "redpanda.SecretRef.IsValid" (dict "a" (list $tsc.accessKey)))) "r")) -}} {{- $envvars = (concat (default (list) $envvars) (list (mustMergeOverwrite (dict "name" "") (dict "name" "REDPANDA_CLOUD_STORAGE_ACCESS_KEY" "valueFrom" (get (fromJson (include "redpanda.SecretRef.AsSource" (dict "a" (list $tsc.accessKey)))) "r"))))) -}} @@ -1507,12 +1507,12 @@ {{- $c := (index .a 0) -}} {{- range $_ := (list 1) -}} {{- $_is_returning := false -}} -{{- $_2182___containerExists := (get (fromJson (include "_shims.dicttest" (dict "a" (list $c "cloud_storage_azure_container" (coalesce nil))))) "r") -}} -{{- $_ := (index $_2182___containerExists 0) -}} -{{- $containerExists := (index $_2182___containerExists 1) -}} -{{- $_2183___accountExists := (get (fromJson (include "_shims.dicttest" (dict "a" (list $c "cloud_storage_azure_storage_account" (coalesce nil))))) "r") -}} -{{- $_ := (index $_2183___accountExists 0) -}} -{{- $accountExists := (index $_2183___accountExists 1) -}} +{{- $_2187___containerExists := (get (fromJson (include "_shims.dicttest" (dict "a" (list $c "cloud_storage_azure_container" (coalesce nil))))) "r") -}} +{{- $_ := (index $_2187___containerExists 0) -}} +{{- $containerExists := (index $_2187___containerExists 1) -}} +{{- $_2188___accountExists := (get (fromJson (include "_shims.dicttest" (dict "a" (list $c "cloud_storage_azure_storage_account" (coalesce nil))))) "r") -}} +{{- $_ := (index $_2188___accountExists 0) -}} +{{- $accountExists := (index $_2188___accountExists 1) -}} {{- $_is_returning = true -}} {{- (dict "r" (and $containerExists $accountExists)) | toJson -}} {{- break -}} @@ -1523,9 +1523,9 @@ {{- $c := (index .a 0) -}} {{- range $_ := (list 1) -}} {{- $_is_returning := false -}} -{{- $_2188_value_ok := (get (fromJson (include "_shims.dicttest" (dict "a" (list $c `cloud_storage_cache_size` (coalesce nil))))) "r") -}} -{{- $value := (index $_2188_value_ok 0) -}} -{{- $ok := (index $_2188_value_ok 1) -}} +{{- $_2193_value_ok := (get (fromJson (include "_shims.dicttest" (dict "a" (list $c `cloud_storage_cache_size` (coalesce nil))))) "r") -}} +{{- $value := (index $_2193_value_ok 0) -}} +{{- $ok := (index $_2193_value_ok 1) -}} {{- if (not $ok) -}} {{- $_is_returning = true -}} {{- (dict "r" (coalesce nil)) | toJson -}} diff --git a/charts/redpanda/chart/values.schema.json b/charts/redpanda/chart/values.schema.json index 9deecf6c7..af6324a63 100644 --- a/charts/redpanda/chart/values.schema.json +++ b/charts/redpanda/chart/values.schema.json @@ -21893,6 +21893,9 @@ "pvcUnbinder": { "additionalProperties": false, "properties": { + "disableStuckClaimExemption": { + "type": "boolean" + }, "enabled": { "type": "boolean" }, diff --git a/charts/redpanda/chart/values.yaml b/charts/redpanda/chart/values.yaml index 4ad183fbb..bad62c44b 100644 --- a/charts/redpanda/chart/values.yaml +++ b/charts/redpanda/chart/values.yaml @@ -706,8 +706,15 @@ statefulset: # such as `local` or `hostPath`, are used. It does so by monitoring redpanda Pods that are in a "Pending" state for at least the period # specified by `unbindAfter`. After this period it will delete the Pod's PVC to re-trigger volume provisioning. pvcUnbinder: + # -- Enables the PVCUnbinder sidecar controller. Note: with `rbac.enabled`, this renders a + # ClusterRole that grants the Pod's ServiceAccount cluster-wide read access to Nodes, + # regardless of the other pvcUnbinder settings. enabled: false unbindAfter: 60s + # -- Renders `--disable-pvc-rebinding-gate-exemption`: turns off the pvc-rebinding gate's + # stuck-claim exemption (an escape hatch if its proof chain misfires in your environment) + # while keeping the rest of the PVCUnbinder running. + disableStuckClaimExemption: false # The BrokerDecommissioner attempts to decommission brokers that have been non-gracefully terminated in a cluster due to something # like a Node crash. When a broker is determined to no longer be valid (i.e. it has been replaced by a broker with a different id) # the decommissioner reaches out to the deployed cluster and begins a decommission operation after detecting the broker as invalid diff --git a/charts/redpanda/statefulset.go b/charts/redpanda/statefulset.go index 27f050434..8f5d8585b 100644 --- a/charts/redpanda/statefulset.go +++ b/charts/redpanda/statefulset.go @@ -1191,6 +1191,10 @@ func statefulSetContainerSidecar(state *RenderState, pool Pool) *corev1.Containe `--run-pvc-unbinder`, fmt.Sprintf("--pvc-unbinder-timeout=%s", pool.Statefulset.SideCars.PVCUnbinder.UnbindAfter), }...) + + if pool.Statefulset.SideCars.PVCUnbinder.DisableStuckClaimExemption { + args = append(args, `--disable-pvc-rebinding-gate-exemption`) + } } args = append(args, pool.Statefulset.SideCars.Args...) diff --git a/charts/redpanda/testdata/template-cases.txtar b/charts/redpanda/testdata/template-cases.txtar index 08b1ff024..de76b1ce5 100644 --- a/charts/redpanda/testdata/template-cases.txtar +++ b/charts/redpanda/testdata/template-cases.txtar @@ -1185,6 +1185,15 @@ statefulset: pvcUnbinder: enabled: false +-- pvc-unbinder-disable-stuck-claim-exemption -- +# ASSERT-NO-ERROR +# ASSERT-FIELD-CONTAINS ["apps/v1/StatefulSet", "default/redpanda", "{.spec.template.spec.containers[?(@.name == \"sidecar\")].args}", "--disable-pvc-rebinding-gate-exemption"] +statefulset: + sideCars: + pvcUnbinder: + enabled: true + disableStuckClaimExemption: true + -- jit-certificates -- # ASSERT-NO-ERROR # ASSERT-FIELD-EQUALS ["apps/v1/StatefulSet", "default/redpanda", "{.spec.template.spec.volumes[?(@.name == \"redpanda-external-cert\")]}", {"name": "redpanda-external-cert", "emptyDir": {}}] diff --git a/charts/redpanda/values.go b/charts/redpanda/values.go index 5c019f67a..54f212e43 100644 --- a/charts/redpanda/values.go +++ b/charts/redpanda/values.go @@ -1039,6 +1039,11 @@ type Sidecars struct { PVCUnbinder struct { Enabled bool `json:"enabled"` UnbindAfter string `json:"unbindAfter"` + // DisableStuckClaimExemption renders the sidecar flag + // --disable-pvc-rebinding-gate-exemption, turning off the + // pvc-rebinding gate's stuck-claim exemption while keeping the + // rest of the PVCUnbinder running. + DisableStuckClaimExemption bool `json:"disableStuckClaimExemption"` } `json:"pvcUnbinder"` BrokerDecommissioner struct { Enabled bool `json:"enabled"` diff --git a/charts/redpanda/values_partial.gen.go b/charts/redpanda/values_partial.gen.go index 523f6bee7..724b39941 100644 --- a/charts/redpanda/values_partial.gen.go +++ b/charts/redpanda/values_partial.gen.go @@ -290,8 +290,9 @@ type PartialSidecars struct { Image *PartialImage "json:\"image,omitempty\"" Args []string "json:\"args,omitempty\"" PVCUnbinder *struct { - Enabled *bool "json:\"enabled,omitempty\"" - UnbindAfter *string "json:\"unbindAfter,omitempty\"" + Enabled *bool "json:\"enabled,omitempty\"" + UnbindAfter *string "json:\"unbindAfter,omitempty\"" + DisableStuckClaimExemption *bool "json:\"disableStuckClaimExemption,omitempty\"" } "json:\"pvcUnbinder,omitempty\"" BrokerDecommissioner *struct { Enabled *bool "json:\"enabled,omitempty\"" diff --git a/operator/chart/files/rbac/pvcunbinder.ClusterRole.yaml b/operator/chart/files/rbac/pvcunbinder.ClusterRole.yaml index a969b3fbf..a70f0fdd1 100644 --- a/operator/chart/files/rbac/pvcunbinder.ClusterRole.yaml +++ b/operator/chart/files/rbac/pvcunbinder.ClusterRole.yaml @@ -10,6 +10,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -50,3 +51,9 @@ rules: - get - list - watch + - apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get diff --git a/operator/chart/testdata/template-cases.golden.txtar b/operator/chart/testdata/template-cases.golden.txtar index 44c2655e5..4619ebf42 100644 --- a/operator/chart/testdata/template-cases.golden.txtar +++ b/operator/chart/testdata/template-cases.golden.txtar @@ -237,6 +237,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -277,6 +278,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -705,6 +712,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -745,6 +753,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -1150,6 +1164,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -1190,6 +1205,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -1842,6 +1863,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -1882,6 +1904,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -2310,6 +2338,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -2350,6 +2379,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -2782,6 +2817,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -2822,6 +2858,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -3494,6 +3536,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -3534,6 +3577,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -3962,6 +4011,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -4002,6 +4052,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -4420,6 +4476,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -4460,6 +4517,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -5118,6 +5181,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -5158,6 +5222,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -5744,6 +5814,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -5784,6 +5855,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -6247,6 +6324,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -6287,6 +6365,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -7202,6 +7286,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -7242,6 +7327,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -7670,6 +7761,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -7710,6 +7802,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -8129,6 +8227,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -8169,6 +8268,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -8825,6 +8930,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -8865,6 +8971,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -9293,6 +9405,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -9333,6 +9446,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -9790,6 +9909,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -9830,6 +9950,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -10574,6 +10700,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -10614,6 +10741,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -11042,6 +11175,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -11082,6 +11216,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -11585,6 +11725,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -11625,6 +11766,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -12402,6 +12549,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -12442,6 +12590,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -12872,6 +13026,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -12912,6 +13067,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -13346,6 +13507,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -13386,6 +13548,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -14045,6 +14213,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -14085,6 +14254,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -14513,6 +14688,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -14553,6 +14729,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -14977,6 +15159,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -15017,6 +15200,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -15678,6 +15867,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -15718,6 +15908,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -16146,6 +16342,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -16186,6 +16383,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -16642,6 +16845,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -16682,6 +16886,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -17341,6 +17551,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -17381,6 +17592,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -18113,6 +18330,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -18153,6 +18371,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -18826,6 +19050,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -18866,6 +19091,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -19295,6 +19526,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -19335,6 +19567,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -19765,6 +20003,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -19805,6 +20044,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -20482,6 +20727,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -20522,6 +20768,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -21111,6 +21363,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -21151,6 +21404,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -21628,6 +21887,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -21668,6 +21928,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -22494,6 +22760,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -22534,6 +22801,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -23312,6 +23585,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -23352,6 +23626,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -23999,6 +24279,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -24039,6 +24320,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -24468,6 +24755,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -24508,6 +24796,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -24940,6 +25234,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -24980,6 +25275,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -25647,6 +25948,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -25687,6 +25989,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -26115,6 +26423,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -26155,6 +26464,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -26580,6 +26895,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -26620,6 +26936,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -27654,6 +27976,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -27694,6 +28017,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -28125,6 +28454,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -28165,6 +28495,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -28804,6 +29140,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -28844,6 +29181,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -29512,6 +29855,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -29552,6 +29896,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -29980,6 +30330,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -30020,6 +30371,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -30484,6 +30841,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -30524,6 +30882,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -31183,6 +31547,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -31223,6 +31588,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -31809,6 +32180,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -31849,6 +32221,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -32272,6 +32650,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -32312,6 +32691,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -33508,6 +33893,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -33548,6 +33934,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -33982,6 +34374,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -34022,6 +34415,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -34596,6 +34995,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -34636,6 +35036,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -35297,6 +35703,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -35337,6 +35744,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -35924,6 +36337,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -35964,6 +36378,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -36625,6 +37045,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -36665,6 +37086,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -37490,6 +37917,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -37530,6 +37958,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -37958,6 +38392,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -37998,6 +38433,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -38435,6 +38876,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -38475,6 +38917,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -39151,6 +39599,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -39191,6 +39640,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -40405,6 +40860,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -40445,6 +40901,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -41623,6 +42085,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -41663,6 +42126,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -42759,6 +43228,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -42799,6 +43269,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -43886,6 +44362,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -43926,6 +44403,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -44355,6 +44838,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -44395,6 +44879,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -45046,6 +45536,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -45086,6 +45577,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -45764,6 +46261,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -45804,6 +46302,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -46234,6 +46738,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -46274,6 +46779,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -46910,6 +47421,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -46950,6 +47462,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -47619,6 +48137,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -47659,6 +48178,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -48794,6 +49319,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -48834,6 +49360,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -50349,6 +50881,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -50389,6 +50922,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -50819,6 +51358,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -50859,6 +51399,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -51679,6 +52225,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -51719,6 +52266,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -52497,6 +53050,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -52537,6 +53091,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -53464,6 +54024,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -53504,6 +54065,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -54325,6 +54892,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -54365,6 +54933,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -54795,6 +55369,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -54835,6 +55410,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -55724,6 +56305,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -55764,6 +56346,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -58004,6 +58592,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -58044,6 +58633,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -59507,6 +60102,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -59547,6 +60143,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -61336,6 +61938,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -61376,6 +61979,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -62575,6 +63184,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -62615,6 +63225,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -64980,6 +65596,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -65020,6 +65637,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -65448,6 +66071,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -65488,6 +66112,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -66457,6 +67087,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -66497,6 +67128,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -67274,6 +67911,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -67314,6 +67952,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -67742,6 +68386,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -67782,6 +68427,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -68788,6 +69439,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -68828,6 +69480,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -70277,6 +70935,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -70317,6 +70976,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -71510,6 +72175,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -71550,6 +72216,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -72925,6 +73597,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -72965,6 +73638,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -74395,6 +75074,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -74435,6 +75115,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -75391,6 +76077,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -75431,6 +76118,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -76675,6 +77368,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -76715,6 +77409,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -77523,6 +78223,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -77563,6 +78264,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -78153,6 +78860,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -78193,6 +78901,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -79047,6 +79761,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -79087,6 +79802,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -80022,6 +80743,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -80062,6 +80784,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -80495,6 +81223,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -80535,6 +81264,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -81449,6 +82184,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -81489,6 +82225,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -82269,6 +83011,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -82309,6 +83052,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -82737,6 +83486,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -82777,6 +83527,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -83183,6 +83939,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -83223,6 +83980,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -83875,6 +84638,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -83915,6 +84679,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -84343,6 +85113,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -84383,6 +85154,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -84831,6 +85608,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -84871,6 +85649,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -85614,6 +86398,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -85654,6 +86439,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -86082,6 +86873,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -86122,6 +86914,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -86570,6 +87368,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -86610,6 +87409,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -87352,6 +88157,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -87392,6 +88198,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -87820,6 +88632,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -87860,6 +88673,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -88264,6 +89083,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -88304,6 +89124,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -88956,6 +89782,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -88996,6 +89823,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -89424,6 +90257,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -89464,6 +90298,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -89885,6 +90725,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -89925,6 +90766,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -90577,6 +91424,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -90617,6 +91465,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -91045,6 +91899,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -91085,6 +91940,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -91489,6 +92350,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -91529,6 +92391,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -92181,6 +93049,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -92221,6 +93090,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -92649,6 +93524,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -92689,6 +93565,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -93093,6 +93975,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -93133,6 +94016,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -93785,6 +94674,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -93825,6 +94715,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -94253,6 +95149,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -94293,6 +95190,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -94697,6 +95600,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -94737,6 +95641,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -95389,6 +96299,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -95429,6 +96340,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -95857,6 +96774,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -95897,6 +96815,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -96309,6 +97233,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -96349,6 +97274,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -97001,6 +97932,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -97041,6 +97973,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -97469,6 +98407,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -97509,6 +98448,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -97921,6 +98866,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -97961,6 +98907,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -98613,6 +99565,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -98653,6 +99606,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -99081,6 +100040,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -99121,6 +100081,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -99559,6 +100525,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -99599,6 +100566,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -100251,6 +101224,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -100291,6 +101265,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -100719,6 +101699,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -100759,6 +101740,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -101287,6 +102274,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -101327,6 +102315,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -101979,6 +102973,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -102019,6 +103014,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -102447,6 +103448,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -102487,6 +103489,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -102927,6 +103935,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -102967,6 +103976,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -103619,6 +104634,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -103659,6 +104675,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -104087,6 +105109,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -104127,6 +105150,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -104567,6 +105596,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -104607,6 +105637,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -105259,6 +106295,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -105299,6 +106336,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -105727,6 +106770,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -105767,6 +106811,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -106206,6 +107256,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -106246,6 +107297,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -106914,6 +107971,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -106954,6 +108012,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -107382,6 +108446,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -107422,6 +108487,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -107939,6 +109010,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -107979,6 +109051,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -108738,6 +109816,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -108778,6 +109857,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -109206,6 +110291,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -109246,6 +110332,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -109818,6 +110910,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -109858,6 +110951,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -110617,6 +111716,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -110657,6 +111757,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -111085,6 +112191,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -111125,6 +112232,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -111691,6 +112804,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -111731,6 +112845,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -112490,6 +113610,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -112530,6 +113651,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -112958,6 +114085,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -112998,6 +114126,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -113497,6 +114631,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -113537,6 +114672,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -114296,6 +115437,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -114336,6 +115478,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -114764,6 +115912,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -114804,6 +115953,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -115295,6 +116450,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -115335,6 +116491,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -116078,6 +117240,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -116118,6 +117281,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -116546,6 +117715,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -116586,6 +117756,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -117121,6 +118297,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -117161,6 +118338,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -117813,6 +118996,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -117853,6 +119037,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -118281,6 +119471,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -118321,6 +119512,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -118732,6 +119929,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -118772,6 +119970,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -119424,6 +120628,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -119464,6 +120669,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -119892,6 +121103,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -119932,6 +121144,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: @@ -120337,6 +121555,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -120377,6 +121596,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get - apiGroups: - "" resources: diff --git a/operator/cmd/multicluster/multicluster.go b/operator/cmd/multicluster/multicluster.go index 54f0622f7..de5ce09ea 100644 --- a/operator/cmd/multicluster/multicluster.go +++ b/operator/cmd/multicluster/multicluster.go @@ -96,6 +96,7 @@ type MulticlusterOptions struct { UnbindPVCsAfter time.Duration AllowPVRebinding bool + DisablePVCRebindingGateExemption bool UnbinderSelector pflagutil.LabelSelectorValue BrokerPodNodeUnavailableToleration time.Duration @@ -231,6 +232,7 @@ func (o *MulticlusterOptions) BindFlags(cmd *cobra.Command) { cmd.Flags().DurationVar(&o.UnbindPVCsAfter, "unbind-pvcs-after", 0, "if not zero, runs the PVCUnbinder controller which attempts to 'unbind' the PVCs' of Pods that are Pending for longer than the given duration") cmd.Flags().BoolVar(&o.AllowPVRebinding, "allow-pv-rebinding", false, "DEPRECATED. When the PVCUnbinder fires, also clear the freed PV's ClaimRef so the disk can be reused if the node returns. Risks cross-broker disk swap when multiple PVs are cleared concurrently. Leave at false (the default) — the unbinder's pause-annotation, multi-pod auto-detect, and per-cluster serialization gates make this flag unnecessary in practice and unsafe in the cases where it would have been useful.") _ = cmd.Flags().MarkDeprecated("allow-pv-rebinding", "the gating checks added to the PVCUnbinder (pause annotation, multi-pod auto-detect, per-cluster serialization) supersede this flag; see the PVCUnbinder controller godoc") + cmd.Flags().BoolVar(&o.DisablePVCRebindingGateExemption, "disable-pvc-rebinding-gate-exemption", false, "Escape hatch: turn off the PVCUnbinder's stuck-claim exemption so its pvc-rebinding gate defers on every unbound claim (the pre-exemption behavior). Use if the exemption's proof chain misfires in your environment; unlike the pause annotation it keeps the rest of the unbinder running.") cmd.Flags().Var(&o.UnbinderSelector, "unbinder-label-selector", "if provided, a Kubernetes label selector that will filter Pods to be considered by the PVCUnbinder.") cmd.Flags().DurationVar(&o.BrokerPodNodeUnavailableToleration, "broker-pod-node-unavailable-toleration", 0, "Controls injection of node.kubernetes.io/not-ready and node.kubernetes.io/unreachable NoExecute tolerations onto broker pods. 0 (default) = feature off, no tolerations injected. Positive = tolerationSeconds set to this duration. Negative (-1s or any negative value) = tolerate forever, no tolerationSeconds field (appropriate for cloud K8s where Node-object deletion is the authoritative signal of permanent node loss). User-set tolerations for these taint keys are always preserved.") cmd.Flags().DurationVar(&o.ClusterConnectionTimeout, "cluster-connection-timeout", 10*time.Second, "Timeout for internal clients used to connect to Redpanda clusters (admin API in particular)") @@ -506,10 +508,11 @@ func Run( } else { setupLog.Info("starting PVCUnbinder controller", "unbind-after", opts.UnbindPVCsAfter, "selector", opts.UnbinderSelector, "allow-pv-rebinding", opts.AllowPVRebinding) if err := (&pvcunbinder.MulticlusterController{ - Manager: manager, - Timeout: opts.UnbindPVCsAfter, - Selector: opts.UnbinderSelector.Selector, - AllowRebinding: opts.AllowPVRebinding, + Manager: manager, + Timeout: opts.UnbindPVCsAfter, + Selector: opts.UnbinderSelector.Selector, + AllowRebinding: opts.AllowPVRebinding, + DisableStuckClaimExemption: opts.DisablePVCRebindingGateExemption, }).SetupWithMultiClusterManager(); err != nil { setupLog.Error(err, "unable to create controller", "controller", "PVCUnbinder") return err diff --git a/operator/cmd/run/run.go b/operator/cmd/run/run.go index 96e33f54f..8755b414e 100644 --- a/operator/cmd/run/run.go +++ b/operator/cmd/run/run.go @@ -127,6 +127,7 @@ type RunOptions struct { unbindPVCsAfter time.Duration unbinderSelector pflagutil.LabelSelectorValue allowPVRebinding bool + disablePVCRebindingGateExemption bool brokerPodNodeUnavailableToleration time.Duration autoDeletePVCs bool webhookCertPath string @@ -204,6 +205,7 @@ func (o *RunOptions) BindFlags(cmd *cobra.Command) { cmd.Flags().DurationVar(&o.unbindPVCsAfter, "unbind-pvcs-after", 0, "if not zero, runs the PVCUnbinder controller which attempts to 'unbind' the PVCs' of Pods that are Pending for longer than the given duration; the Broker controller's dead-node PV remediation (--enable-broker) also uses this timeout when set, and falls back to its own default (5m) when zero") cmd.Flags().BoolVar(&o.allowPVRebinding, "allow-pv-rebinding", false, "DEPRECATED. When the PVCUnbinder fires, also clear the freed PV's ClaimRef so the disk can be reused if the node returns. Risks cross-broker disk swap when multiple PVs are cleared concurrently. Leave at false (the default) — the unbinder's pause-annotation, multi-pod auto-detect, and per-cluster serialization gates make this flag unnecessary in practice and unsafe in the cases where it would have been useful.") _ = cmd.Flags().MarkDeprecated("allow-pv-rebinding", "the gating checks added to the PVCUnbinder (pause annotation, multi-pod auto-detect, per-cluster serialization) supersede this flag; see the PVCUnbinder controller godoc") + cmd.Flags().BoolVar(&o.disablePVCRebindingGateExemption, "disable-pvc-rebinding-gate-exemption", false, "Escape hatch: turn off the PVCUnbinder's stuck-claim exemption so its pvc-rebinding gate defers on every unbound claim (the pre-exemption behavior). Use if the exemption's proof chain misfires in your environment; unlike the pause annotation it keeps the rest of the unbinder running.") cmd.Flags().Var(&o.unbinderSelector, "unbinder-label-selector", "if provided, a Kubernetes label selector that will filter Pods to be considered by the PVCUnbinder.") cmd.Flags().DurationVar(&o.brokerPodNodeUnavailableToleration, "broker-pod-node-unavailable-toleration", 0, "Controls injection of node.kubernetes.io/not-ready and node.kubernetes.io/unreachable NoExecute tolerations onto broker pods. 0 (default) = feature off, no tolerations injected. Positive = tolerationSeconds set to this duration. Negative (-1s or any negative value) = tolerate forever, no tolerationSeconds field (appropriate for cloud K8s where Node-object deletion is the authoritative signal of permanent node loss). User-set tolerations for these taint keys are always preserved.") cmd.Flags().BoolVar(&o.autoDeletePVCs, "auto-delete-pvcs", false, "Use StatefulSet PersistentVolumeClaimRetentionPolicy to auto delete PVCs on scale down and Cluster resource delete.") @@ -645,10 +647,11 @@ func Run( setupLog.Info("starting PVCUnbinder controller", "unbind-after", opts.unbindPVCsAfter, "selector", opts.unbinderSelector, "allow-pv-rebinding", opts.allowPVRebinding) if err := (&pvcunbinder.Controller{ - Client: mgr.GetClient(), - Timeout: opts.unbindPVCsAfter, - Selector: opts.unbinderSelector.Selector, - AllowRebinding: opts.allowPVRebinding, + Client: mgr.GetClient(), + Timeout: opts.unbindPVCsAfter, + Selector: opts.unbinderSelector.Selector, + AllowRebinding: opts.allowPVRebinding, + DisableStuckClaimExemption: opts.disablePVCRebindingGateExemption, }).SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to create controller", "controller", "PVCUnbinder") return err diff --git a/operator/cmd/sidecar/sidecar.go b/operator/cmd/sidecar/sidecar.go index 8c92a1d77..6a2c15618 100644 --- a/operator/cmd/sidecar/sidecar.go +++ b/operator/cmd/sidecar/sidecar.go @@ -69,6 +69,7 @@ func Command() *cobra.Command { brokerProbeBrokerURL string runUnbinder bool unbinderTimeout time.Duration + unbinderDisableExemption bool selector pflagutil.LabelSelectorValue panicAfter time.Duration ) @@ -102,6 +103,7 @@ func Command() *cobra.Command { brokerProbeBrokerURL, runUnbinder, unbinderTimeout, + unbinderDisableExemption, selector.Selector, panicAfter, ) @@ -146,6 +148,7 @@ func Command() *cobra.Command { // unbinder flags cmd.Flags().BoolVar(&runUnbinder, "run-pvc-unbinder", false, "Specifies if the PVC unbinder should be run.") cmd.Flags().DurationVar(&unbinderTimeout, "pvc-unbinder-timeout", 60*time.Second, "The time period to wait before removing any unbound PVCs.") + cmd.Flags().BoolVar(&unbinderDisableExemption, "disable-pvc-rebinding-gate-exemption", false, "Escape hatch: turn off the PVC unbinder's stuck-claim exemption so its pvc-rebinding gate defers on every unbound claim (the pre-exemption behavior).") // Internal use flags. cmd.Flags().DurationVar(&panicAfter, "panic-after", 0, "If non-zero, will trigger an unhandled panic after the specified time resulting in a process crash.") @@ -177,6 +180,7 @@ func Run( brokerProbeBrokerURL string, runUnbinder bool, unbinderTimeout time.Duration, + unbinderDisableExemption bool, selector labels.Selector, panicAfter time.Duration, ) error { @@ -270,9 +274,10 @@ func Run( setupLog.Info("PVC unbinder enabled", "namespace", clusterNamespace, "selector", selector) if err := (&pvcunbinder.Controller{ - Client: mgr.GetLocalManager().GetClient(), - Timeout: unbinderTimeout, - Selector: selector, + Client: mgr.GetLocalManager().GetClient(), + Timeout: unbinderTimeout, + Selector: selector, + DisableStuckClaimExemption: unbinderDisableExemption, }).SetupWithManager(mgr.GetLocalManager()); err != nil { setupLog.Error(err, "unable to create controller", "controller", "PVCUnbinder") return err diff --git a/operator/config/rbac/bases/operator/role.yaml b/operator/config/rbac/bases/operator/role.yaml index 16fcc388b..4210cae4d 100644 --- a/operator/config/rbac/bases/operator/role.yaml +++ b/operator/config/rbac/bases/operator/role.yaml @@ -349,6 +349,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role diff --git a/operator/config/rbac/itemized/pvcunbinder.yaml b/operator/config/rbac/itemized/pvcunbinder.yaml index 4dcfbea5b..5ac9809b5 100644 --- a/operator/config/rbac/itemized/pvcunbinder.yaml +++ b/operator/config/rbac/itemized/pvcunbinder.yaml @@ -10,6 +10,7 @@ rules: - nodes verbs: - get + - list - apiGroups: - "" resources: @@ -50,6 +51,12 @@ rules: - get - list - watch +- apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role diff --git a/operator/internal/controller/pvcunbinder/pvcunbinder.go b/operator/internal/controller/pvcunbinder/pvcunbinder.go index f1176a720..6d93dfd7d 100644 --- a/operator/internal/controller/pvcunbinder/pvcunbinder.go +++ b/operator/internal/controller/pvcunbinder/pvcunbinder.go @@ -18,6 +18,7 @@ import ( "time" corev1 "k8s.io/api/core/v1" + storagev1 "k8s.io/api/storage/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -47,25 +48,35 @@ import ( // scenarios. var SchedulingFailureRE = regexp.MustCompile(`(^0/[1-9]\d* nodes are available)|(volume node affinity)`) -// PauseAnnotation - when present and set to "true" on the parent -// Redpanda or Cluster CR, instructs the PVCUnbinder to skip all -// reconcile work for any Pod that belongs to that cluster. Used by the -// cloud control plane (and operators in general) to pause unbinder -// activity around planned events like K8s cluster upgrades, node-pool -// surges, or maintenance windows where transient multi-node disruption -// is expected. +// PauseAnnotation pauses the PVCUnbinder for a whole cluster. Set it +// to "true" on the parent Redpanda or Cluster CR. Use it during +// planned events (cluster upgrades, node-pool surges, maintenance) +// when many nodes are disrupted at once. const PauseAnnotation = "operator.redpanda.com/pause-pvc-unbinder" -// requeueDuringDisruption is how long to wait before re-checking when -// we've decided to skip an unbind action (paused via annotation, -// concurrent K8s-wide disruption detected, or another sibling unbind -// in flight). +// requeueDuringDisruption is the wait between re-checks after any +// gate defers an unbind. const requeueDuringDisruption = 30 * time.Second -// Gate names used as label values on the -// `operator_controller_pvc_unbinder_gate_deferred_total` metric and as -// the Event Reason recorded on the Pod whose remediation was deferred. -// Keep this list closed — these are the only allowed values. +// The unbinder runs five safety gates before it deletes anything. +// Each gate can defer (postpone) the unbind. In order: +// +// - Gate 0 "in-flight": a previous unbind in this cluster has not +// finished yet (its recreated claim is not bound). Wait for it. +// - Gate 1 "pause": the cluster CR carries [PauseAnnotation]. Wait. +// - Gate 2 "multi-node": stuck pods are pinned to several different +// nodes, which looks like a cluster-wide event, not a single node +// failure. Wait. +// - Gate 3 "pvc-rebinding": some claim in the cluster is not bound +// yet. It is probably re-binding right now, so wait — unless the +// claim is exempt because its pod is provably deadlocked (see +// [Controller.stuckClaimNames]). +// - Gate 4 "freed-pv": a PV freed by --allow-pv-rebinding is still +// floating and could pair with the wrong claim. Wait. +// +// These names are the label values on the +// `..._pvc_unbinder_gate_deferred_total` metric and appear in the +// deferral Event on the Pod. Keep this list closed. const ( gateInFlight = "in-flight" gatePause = "pause" @@ -74,103 +85,85 @@ const ( gateFreedPV = "freed-pv" ) -// The unbinder records its in-flight state as annotations on the -// PersistentVolumes it operates on, NOT in process memory. PVs are -// forced to a Retain policy before any destructive action and survive -// operator restarts, leader handoffs, and the pod/PVC deletions that -// make up an unbind — so every gate that reads these annotations is -// crash-safe by construction. All reads of these annotations go -// through the uncached Reader: the annotations are written by this -// controller moments before they're needed, which is exactly the -// window where the informer cache lags. +// The unbinder stores its progress as annotations on the PVs it works +// on, not in memory. PVs survive operator restarts and the deletions +// that make up an unbind, so the gates that read these annotations +// are crash-safe. All reads go through the uncached Reader, because +// the annotations are written moments before they are read — exactly +// when the informer cache lags. const ( // InFlightAnnotation marks a PV whose bound PVC this controller is - // about to delete (written in the same patch that forces the - // Retain policy, BEFORE the PVC delete). The value is the cluster - // key of the Redpanda cluster the PV belongs to. + // about to delete. It is written together with the Retain policy, + // before the delete. The value is the cluster key. // - // While any PV carries this annotation for a cluster, Gate 0 - // defers all further unbinds in that cluster. The annotation is - // cleared once the deleted claim is observed recreated (same - // name, different UID) and bound — i.e., the previous unbind has - // fully settled. + // While any PV in a cluster carries this annotation, Gate 0 defers + // all further unbinds there. It is cleared once the deleted claim + // is seen recreated (same name, new UID) and bound. InFlightAnnotation = "operator.redpanda.com/pvc-unbinder-in-flight" // InFlightClaimAnnotation records the claim the PV served at - // unbind time as "namespace/name/uid". Written and cleared - // together with InFlightAnnotation. The UID lets the settle check - // distinguish the recreated claim from the not-yet-deleted old - // one; the namespace/name survive the rebinding path nil-ing out - // pv.Spec.ClaimRef. + // unbind time, as "namespace/name/uid". Written and cleared with + // InFlightAnnotation. The UID tells the recreated claim apart from + // the old one that is still being deleted. InFlightClaimAnnotation = "operator.redpanda.com/pvc-unbinder-claim" // FreedPVAnnotation marks a PV whose ClaimRef this controller - // cleared (the `--allow-pv-rebinding` path). The value is the - // cluster key of the Redpanda cluster the PV belonged to. + // cleared (the --allow-pv-rebinding path). The value is the + // cluster key. // - // While a PV carrying this annotation is in Available phase AND - // its pinned node still exists, the unbinder refuses to unbind - // anything else in the same cluster (Gate 4). Rationale: a freed - // Available PV is a first-class binding candidate for ANY new PVC - // — if we unbind a second broker while the first broker's freed - // disk is still floating, the scheduler can pair the second - // broker's fresh claim with the first broker's old disk (the - // cross-broker swap from INC-2818, reproduced sequentially even - // with serialized unbinds). The annotation is cleared once the PV - // is observed Bound again. + // While such a PV is Available and its pinned node still exists, + // Gate 4 blocks further unbinds in the same cluster. Reason: an + // Available PV can bind to ANY new claim, so unbinding a second + // broker while the first broker's freed disk still floats can give + // the second broker the first broker's disk (the INC-2818 + // cross-broker swap). Cleared once the PV is Bound again. FreedPVAnnotation = "operator.redpanda.com/pvc-unbinder-freed" ) -// eventReasonGateDeferred is the EventReason emitted on the Pod when a -// safety gate defers remediation. Operators watching for "why isn't -// the PVCUnbinder acting on my stuck pod" can `kubectl describe pod` -// and see this reason + the gate label. +// eventReasonGateDeferred is the Event reason written on the Pod when +// a gate defers remediation. `kubectl describe pod` shows it together +// with the gate name. const eventReasonGateDeferred = "PVCUnbinderDeferred" -// Gate 2 identifies "Redpanda broker pod" via two label sets, because -// the cluster types don't share a single pod label that uniquely marks -// Redpanda brokers: +// eventReasonGateExempted is the Event reason written on the Pod when +// Gate 3 is passed because every unbound claim was exempted. A gate +// override gets the same paper trail as a deferral (metric, Warning +// Event, log) so incidents stay easy to attribute. +const eventReasonGateExempted = "PVCUnbinderGateExempted" + +// Gate 2 finds Redpanda broker pods with two label queries, because +// no single pod label covers all cluster types: // -// - v1 Cluster (operator): `app.kubernetes.io/managed-by=redpanda-operator` -// (hardcoded at operator/pkg/labels/labels.go). v1 pods do NOT -// carry `cluster.redpanda.com/broker`. -// - v2 Redpanda, StretchCluster, and direct Helm installs all render -// broker pods through the redpanda chart, whose pod template sets -// `cluster.redpanda.com/broker=true` (charts/redpanda/statefulset.go, -// StatefulSetPodLabels). NB: the operator's -// `cluster.redpanda.com/operator=v2` ownership label lands on the -// StatefulSet OBJECT only — it is never propagated to the pod -// template, so it cannot be used to select pods. +// - v1 Cluster pods carry app.kubernetes.io/managed-by=redpanda-operator. +// - v2 Redpanda, StretchCluster, and Helm installs render pods +// through the redpanda chart, which sets +// cluster.redpanda.com/broker=true. (The operator=v2 label exists +// only on the StatefulSet object, never on pods.) // -// Filtering on `app.kubernetes.io/name=redpanda` would catch all of -// these by default but breaks for users running with `nameOverride` -// (which is a supported customization in production). So Gate 2 does -// two LIST queries and unions the results by (namespace, name). +// app.kubernetes.io/name=redpanda would cover both but breaks under +// nameOverride, so Gate 2 runs both queries and unions the results. const ( managedByLabelValue = "redpanda-operator" brokerLabelKey = "cluster.redpanda.com/broker" brokerLabelValue = "true" ) -// Controller is a Kubernetes Controller that watches for Pods stuck in a -// Pending state due to volume affinities and attempts a remediation. +// Controller watches for Pods stuck in Pending because their local +// volume is pinned to a node they can no longer run on, and frees +// them. // -// It watches for Pod events rather than Node events because: -// 1. Node Deletion events could be missed if the operator is scheduled on the node that's died -// 2. We don't want to re-implement label matching. In theory, it should be -// easy but it's quite risky and behaviors could diverge between Kubernetes -// versions. +// It watches Pod events, not Node events: a Node-deletion event can +// be missed when the operator itself ran on the dead node, and +// re-implementing the scheduler's label matching would be risky. // -// To get the Pod to reschedule we: -// 1. Find all PVs and PVCs associated with our Pod. -// 2. Ensure that all PVs in question have a Retain policy -// 3. Delete all PVCs from step 1. (PVCs are immutable after creation, -// deletion is the only option) -// 4. (Optionally) "Recycle" all PVs from step 1 by clearing the ClaimRef. -// Kubernetes will only consider binding PVs that have a satisfiable -// NodeAffinity. By "recycling" we permit Flakey Nodes to rejoin the cluster -// which _might_ reclaim the now freed volume. -// 5. Deleting the Pod to re-trigger PVC creation and rebinding. +// To let a stuck Pod reschedule it: +// 1. finds the Pod's PVs and PVCs, +// 2. sets a Retain policy on those PVs, +// 3. deletes the PVCs (PVCs are immutable; delete is the only way), +// 4. optionally clears the PVs' ClaimRef (--allow-pv-rebinding) so a +// returning node might reclaim its old volume, +// 5. deletes the Pod, which makes the StatefulSet recreate Pod and +// PVCs and bind them somewhere schedulable. type Controller struct { Client client.Client // Timeout is the duration a Pod must be stuck in Pending before @@ -179,35 +172,39 @@ type Controller struct { // Selector, if specified, will narrow the scope of Pods that this // Reconciler will consider for remediation. Selector labels.Selector - // AllowRebinding optionally enables clearing of the unbound PV's ClaimRef - // which effectively makes the PVs "re-bindable" if the underlying Node - // become capable of scheduling Pods once again. - // NOTE: This option can present problems when a Node's name is reused and - // using HostPath volumes and LocalPathProvisioner. In such a case, the - // helper Pod of LocalPathProvisioner will NOT run a second time as the - // Volume is assumed to exist. This can lead to Permission errors or - // referencing a directory that does not exist. + // AllowRebinding also clears the freed PV's ClaimRef so the disk + // can bind again if its node returns. Deprecated and risky: with + // HostPath volumes and node-name reuse it can produce permission + // errors or point at missing directories (LocalPathProvisioner's + // helper Pod does not run again for a volume it believes already + // exists), and it disables the Gate 3 exemption entirely. AllowRebinding bool + // DisableStuckClaimExemption turns off Gate 3's stuck-claim + // exemption and restores the old behavior: defer on every unbound + // claim. It is an escape hatch for environments where the + // exemption's proof chain misfires (for example, unusual local-PV + // node-affinity shapes). Unlike the pause annotation, it keeps the + // rest of the unbinder running. + DisableStuckClaimExemption bool // ClusterName disambiguates cluster keys in multicluster mode. // Empty for single-cluster operation. ClusterName string // Recorder, if non-nil, receives an Event on the Pod every time a - // safety gate defers remediation. Nil-safe — if unset, only the - // metric is incremented. Uses the new k8s.io/client-go/tools/events + // safety gate defers remediation, and a Warning Event when Gate 3 + // is passed via the stuck-claim exemption. Nil-safe — if unset, + // only the metrics are incremented. Uses the new k8s.io/client-go/tools/events // API rather than the deprecated tools/record API. Recorder events.EventRecorder - // Reader is an uncached client.Reader (the manager's APIReader) - // used where reading through the informer cache would defeat the - // purpose of the check: + // Reader is an uncached client.Reader (the manager's APIReader). + // It is used wherever a stale cache would defeat the check: // - // - the Gate 0/4 PV-annotation scans and the Gate 0 claim settle - // check, which read back state this controller wrote moments - // earlier — exactly the window where the informer cache lags — - // and whose durability guarantee depends on seeing true - // API-server state, and - // - Node existence checks in the freed-PV gate, where a cached - // Get would force the informer to watch every Node in the - // cluster for a check that only runs during disruptions. + // - Gate 0/4 annotation scans, which read back state this + // controller wrote moments earlier; + // - Node lookups, so the cache does not have to watch every + // Node for checks that only run during incidents; + // - all Gate 3 exemption evidence (pod re-check, sibling and + // occupant pod lists, PVC and StorageClass reads), because a + // stale read there could wrongly unlock deletion. // // Falls back to Client when nil (tests). Reader client.Reader @@ -226,20 +223,40 @@ func (r *Controller) reader() client.Reader { // MulticlusterController is a multicluster-aware version of Controller that // watches Pods across all clusters managed by a multicluster.Manager. type MulticlusterController struct { - Manager multicluster.Manager - Timeout time.Duration - Selector labels.Selector - AllowRebinding bool + Manager multicluster.Manager + Timeout time.Duration + Selector labels.Selector + AllowRebinding bool + DisableStuckClaimExemption bool } -// recordGateDeferred increments the gate-defer metric and emits a -// Kubernetes Event on the Pod whose reconcile got deferred. The metric -// path always runs (operators rely on it to alert on silent inaction); -// the Event is skipped when Recorder is nil (the test path). -// -// `action` is the new-events-API verb describing what the unbinder -// just did ("Defer"); `gate` is included in the message so operators -// can tell which gate fired from `kubectl describe pod`. +// claimListForEvent renders a claim-name list for an Event note, +// capped by BOTH name count and total rendered length (claim names +// can legally reach 253 characters, so a count cap alone does not +// bound the note). events.k8s.io/v1 rejects notes longer than 1024 +// characters and the events broadcaster silently DROPS the rejected +// Event, so an unbounded list would erase the paper trail in exactly +// the largest incidents. Logs carry the full list. +func claimListForEvent(names []string) string { + const maxNames = 8 + const maxChars = 700 + n, chars := 0, 0 + for _, name := range names { + if n == maxNames || chars+len(name) > maxChars { + break + } + n++ + chars += len(name) + 1 + } + if n == len(names) { + return fmt.Sprintf("%v", names) + } + return fmt.Sprintf("%v (+%d more)", names[:n], len(names)-n) +} + +// recordGateDeferred increments the gate-defer metric and, when a +// Recorder is set, writes an Event on the Pod. The metric always +// runs; operators alert on it to notice silent inaction. func (r *Controller) recordGateDeferred(pod *corev1.Pod, gate, message string) { observability.PVCUnbinderGateDeferred.WithLabelValues(gate).Inc() if r.Recorder != nil && pod != nil { @@ -279,19 +296,21 @@ func (r *MulticlusterController) Reconcile(ctx context.Context, req mcreconcile. } c := &Controller{ - Client: k8sCluster.GetClient(), - Timeout: r.Timeout, - Selector: r.Selector, - AllowRebinding: r.AllowRebinding, - ClusterName: req.ClusterName, - Recorder: k8sCluster.GetEventRecorder("pvc-unbinder"), - Reader: k8sCluster.GetAPIReader(), + Client: k8sCluster.GetClient(), + Timeout: r.Timeout, + Selector: r.Selector, + AllowRebinding: r.AllowRebinding, + DisableStuckClaimExemption: r.DisableStuckClaimExemption, + ClusterName: req.ClusterName, + Recorder: k8sCluster.GetEventRecorder("pvc-unbinder"), + Reader: k8sCluster.GetAPIReader(), } return c.Reconcile(ctx, req.Request) } // +kubebuilder:rbac:groups=core,resources=persistentvolumes,verbs=get;list;watch;patch -// +kubebuilder:rbac:groups=core,resources=nodes,verbs=get +// +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list +// +kubebuilder:rbac:groups=storage.k8s.io,resources=storageclasses,verbs=get // +kubebuilder:rbac:groups=cluster.redpanda.com,resources=redpandas,verbs=get;list;watch // +kubebuilder:rbac:groups=cluster.redpanda.com,resources=stretchclusters,verbs=get;list;watch // +kubebuilder:rbac:groups=redpanda.vectorized.io,resources=clusters,verbs=get;list;watch @@ -335,16 +354,14 @@ func (r *Controller) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr).For(&corev1.Pod{}, builder.WithPredicates(selectorPredicate, unbinderPredicate)).Complete(r) } -// Reconcile implements the algorithm described in the docs of [Controller]. To -// the best of it's ability, Reconcile is implemented to be idempotent. Due to -// the lack of transactions in Kubernetes/etc and the need to operate across -// many objects, it's quite difficult to guarantee this. The general strategy -// is to fetch a snapshot of the world as early as possible and then rely on -// ResourceVersions to inform us about changes from external actors, in which -// case we'll re-queue. Recovery from partial failures relies on the durable -// in-flight PV annotations: any successful prefix of the action steps leaves -// state that Gate 0 either holds on (siblings) or resumes from (the same pod -// retrying — see checkPVGates' own-claim exemption). +// Reconcile runs the algorithm described on [Controller]: it checks +// the five safety gates in order (see the gate constants) and, if all +// pass, performs the unbind steps. It aims to be idempotent. Because +// Kubernetes has no transactions, it takes an early snapshot, guards +// every delete with UID/ResourceVersion preconditions, and re-queues +// on conflicts. If it crashes half-way, the durable in-flight PV +// annotations let Gate 0 hold siblings back and let the same pod +// resume its own unbind. func (r *Controller) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { logger := ctrl.LoggerFrom(ctx).WithName("PVCUnbinder") ctx = log.IntoContext(ctx, logger) @@ -362,23 +379,46 @@ func (r *Controller) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu return ctrl.Result{RequeueAfter: requeueAfter, Requeue: ok}, nil } + // The cached read above is only a cheap pre-filter. Everything + // after this point must be justified by true API-server state. A + // stale cache copy of a Pod that was already recreated or + // scheduled could still look stuck, grant its own exemption, and + // reach the PVC deletes — and the delete preconditions guard the + // claims, not the Pod evidence. So: re-read the Pod uncached, + // qualify it again, and let the fresh object drive the rest. + // + // The re-read decodes into a FRESH object. Decoding into the + // cache-populated one would merge (JSON decode semantics): fields + // the fresh response omits — say, a just-recreated pod's still + // empty status.conditions — would keep their stale cached values + // and defeat the re-qualification below. + var freshPod corev1.Pod + if err := r.reader().Get(ctx, req.NamespacedName, &freshPod); err != nil { + if apierrors.IsNotFound(err) { + return ctrl.Result{}, nil + } + return ctrl.Result{}, err + } + pod = freshPod + if ok, requeueAfter := r.ShouldRemediate(ctx, &pod); !ok || requeueAfter > 0 { + logger.Info("Pod no longer qualifies on the uncached re-read; skipping", "name", pod.Name, "ok", ok, "requeue-after", requeueAfter) + return ctrl.Result{RequeueAfter: requeueAfter, Requeue: ok}, nil + } + // Gates 0 and 4 share one uncached scan over the cluster's - // annotated PVs. Uncached because the annotations are written by - // this controller moments before they're needed — exactly the - // window where the informer cache lags — and because durable - // API-server state (not process memory) is what makes these gates - // survive operator restarts and leader handoffs mid-unbind. + // annotated PVs. Uncached, because the annotations are written + // moments before they are read; durable, so the gates survive + // restarts and leader handoffs mid-unbind. pvGates, err := r.checkPVGates(ctx, r.clusterKey(&pod), &pod) if err != nil { return ctrl.Result{}, err } - // Gate 0: a previous unbind in this cluster has not settled — a PV - // carries the in-flight annotation and its recorded claim has not - // yet been observed recreated (same name, NEW uid) and bound. - // Covers both the deleted-but-not-yet-recreated window and any - // partial failure of a previous reconcile (the annotation is - // written before the first destructive action). + // Gate 0 "in-flight": a previous unbind in this cluster has not + // finished — some PV still carries the in-flight annotation and + // its recorded claim is not yet recreated and bound. This also + // covers partial failures, because the annotation is written + // before the first destructive step. if pvGates.unbindInFlight { const msg = "a previous unbind for this cluster has not settled; deferring" logger.Info(msg, "name", pod.Name) @@ -386,9 +426,8 @@ func (r *Controller) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu return ctrl.Result{RequeueAfter: requeueDuringDisruption}, nil } - // Gate 1: parent CR has the pause annotation set. Operators set this - // around planned events (K8s cluster upgrades, node-pool surges, etc.) - // where transient multi-node disruption is potentially expected. + // Gate 1 "pause": the parent CR carries [PauseAnnotation]. + // Operators set it around planned events like cluster upgrades. if paused, err := r.isClusterPaused(ctx, &pod); err != nil { return ctrl.Result{}, err } else if paused { @@ -398,17 +437,14 @@ func (r *Controller) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu return ctrl.Result{RequeueAfter: requeueDuringDisruption}, nil } - // Gate 2: stuck StatefulSet Pods across the cluster are pinned to - // more than one distinct node. That's the signature of a K8s-wide - // event (cloud control-plane upgrade, AZ hiccup, node-pool surge) - // rather than a single-node failure — defer to natural recovery so - // the unbinder doesn't force fresh PVs / ClaimRef clears on - // brokers spread across multiple failing nodes simultaneously. - // - // Counting distinct nodes (not distinct pods) correctly handles the - // case where multiple co-tenant pods are on the same failed node: - // that's a legitimate single-node failure the unbinder should act - // on, not a K8s-wide event. + // Gate 2 "multi-node": stuck pods are pinned to more than one + // distinct node. That looks like a cluster-wide event (control + // plane upgrade, AZ problem), not a single node failure, so wait + // for natural recovery. Distinct NODES are counted, not pods: + // several pods on one dead node is still a single-node failure + // and the unbinder should act on it. Known gap: two deadlocked + // victims pinned to two different occupied nodes also defer here + // and need manual PVC deletion. if multiNode, err := r.multiNodeEventInProgress(ctx); err != nil { return ctrl.Result{}, err } else if multiNode { @@ -418,34 +454,140 @@ func (r *Controller) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu return ctrl.Result{RequeueAfter: requeueDuringDisruption}, nil } - // Gate 3: a PVC in this cluster is observable but not yet bound - // (empty spec.volumeName). Defer until the binder has re-bound it. - // Gate 0 already covers unbinds *we* performed end-to-end (the - // in-flight annotation isn't cleared until the recreated claim is - // bound); Gate 3 additionally catches unbound claims from external - // actors — e.g. an admin manually deleting a PVC — which carry no - // annotation. Cached read: a false pass here is still backstopped - // by Gate 0 for our own actions, and a false defer is harmless. - clusterPVCsByName, err := r.listClusterPVCsByName(ctx, &pod) + // Gate 3 "pvc-rebinding": some PVC in this cluster is not bound + // yet (empty spec.volumeName). Usually that means a re-bind is in + // progress, so wait. Gate 0 already covers unbinds WE performed; + // this gate also catches unbound claims from external actors, for + // example an admin deleting a PVC by hand. The list is a cached + // read: a false pass is backstopped by Gate 0 for our own actions + // and a false defer only costs 30 seconds. + // + // Exception: claims owned by provably deadlocked Pods are exempt + // (see [Controller.stuckClaimNames]). Waiting on such a claim + // waits forever — under WaitForFirstConsumer it binds only after + // its Pod schedules, and the Pod schedules only after the unbinder + // frees its mis-pinned sibling claim, which is the very action + // this gate would defer. Typical case: a fresh cluster where a + // broker's cache PV landed on a full node; the datadir claim then + // waits forever. Gate 0 still serializes the destructive work. + // + // Under --allow-pv-rebinding there is NO exemption: freed PVs + // float as binding candidates, and acting while any claim is + // unbound could pair it with the wrong disk (INC-2818). + clusterPVCsByName, err := r.listClusterPVCsByName(ctx, r.Client, &pod) if err != nil { return ctrl.Result{}, err } - for _, pvc := range clusterPVCsByName { + var unbound []string + for name, pvc := range clusterPVCsByName { if pvc.Spec.VolumeName == "" { - const msg = "a PVC in this cluster has no volumeName yet; deferring" - logger.Info(msg, "name", pod.Name) + unbound = append(unbound, name) + } + } + // Sorted so that with several unbound claims, consecutive + // reconciles name the SAME gating claim in the log and Event + // (map iteration order would flap the message every 30s, and the + // events API dedups by message content). + slices.Sort(unbound) + if len(unbound) > 0 { + // The exemption evidence runs lazily — only when some claim is + // actually unbound — so the common all-bound path costs no + // extra live reads. If the evidence reads fail (for example a + // 403 when RBAC lags an image upgrade), the error downgrades + // to the conservative deferral instead of error-looping the + // reconcile. That direction is fail-safe: it disables a + // permission, it never grants one. Context cancellation is not + // downgraded; it surfaces as an error. + exemptClaims := map[string]struct{}{} + podMispinned := false + if !r.AllowRebinding && !r.DisableStuckClaimExemption { + if exemptClaims, podMispinned, err = r.stuckClaimNames(ctx, &pod, unbound); err != nil { + if ctxErr := ctx.Err(); ctxErr != nil { + return ctrl.Result{}, ctxErr + } + logger.Error(err, "failed to compute Gate 3 stuck-claim exemptions; keeping the conservative deferral", "name", pod.Name) + exemptClaims = map[string]struct{}{} + } + } + var exempted []string + for _, name := range unbound { + if _, stuck := exemptClaims[name]; stuck { + exempted = append(exempted, name) + continue + } + msg := fmt.Sprintf("PVC %q has no volumeName yet; deferring", name) + logger.Info(msg, "name", pod.Name, "pvc", name) r.recordGateDeferred(&pod, gatePVCRebinding, msg) return ctrl.Result{RequeueAfter: requeueDuringDisruption}, nil } + // Every unbound claim was exempted. Exemptions break the + // stuck-Pod deadlock; they must not authorize destroying an + // unrelated Pod. If the reconciled Pod has no mis-pinned bound + // claim of its own (it is Pending for some other reason, like + // CPU pressure), a sibling's deadlock must not unlock deleting + // this Pod's healthy claims. Both intended victims — the + // deadlocked broker and the dead-node broker — pass this check + // naturally. + if !podMispinned { + logger.Info(fmt.Sprintf("unbound claims %v are exempted, but the reconciled Pod lacks its own mis-pin proof; deferring", exempted), "name", pod.Name) + r.recordGateDeferred(&pod, gatePVCRebinding, fmt.Sprintf("unbound claims %s are exempted, but the reconciled Pod lacks its own mis-pin proof; deferring", claimListForEvent(exempted))) + return ctrl.Result{RequeueAfter: requeueDuringDisruption}, nil + } + // The cached list above can be stale in BOTH directions. + // Extra cached-only unbound claims merely cost a 30s + // deferral, but a LIVE unbound claim the cache has not seen + // yet must not slip past the gate on the exemptions' back. + // Passing the gate is an exemption-granting decision, so it + // is confirmed against an uncached re-list: any live unbound + // claim outside the exempted set defers as usual, and a + // failed re-list defers conservatively (same fail-safe + // direction as the evidence chain). + livePVCsByName, err := r.listClusterPVCsByName(ctx, r.reader(), &pod) + if err != nil { + if ctxErr := ctx.Err(); ctxErr != nil { + return ctrl.Result{}, ctxErr + } + logger.Error(err, "failed to confirm the exempted claims against the live API server; keeping the conservative deferral", "name", pod.Name) + r.recordGateDeferred(&pod, gatePVCRebinding, "uncached PVC re-list failed; deferring") + return ctrl.Result{RequeueAfter: requeueDuringDisruption}, nil + } + liveUnbound := make([]string, 0, len(livePVCsByName)) + for name, pvc := range livePVCsByName { + if pvc.Spec.VolumeName == "" { + liveUnbound = append(liveUnbound, name) + } + } + slices.Sort(liveUnbound) + for _, name := range liveUnbound { + if _, stuck := exemptClaims[name]; !stuck { + msg := fmt.Sprintf("PVC %q has no volumeName on the live API server; deferring", name) + logger.Info(msg, "name", pod.Name, "pvc", name) + r.recordGateDeferred(&pod, gatePVCRebinding, msg) + return ctrl.Result{RequeueAfter: requeueDuringDisruption}, nil + } + } + // A safety gate is being overridden. Leave the same paper + // trail a deferral gets: metric, Event, and log, naming the + // exempted claims. Recorded only when the reconcile really + // proceeds: the freed-pv gate below is durable and can hold + // for days, and counting a "pass" every 30s during that hold + // would poison the metric. The Event is a Warning — it + // precedes destructive deletion, and Warning is what event + // pipelines filter for. + if !pvGates.freedPVUnresolved { + logger.Info(fmt.Sprintf("unbound claims %v are exempted as stuck-Pod claims and the reconciled Pod holds its own mis-pin proof; proceeding past the pvc-rebinding gate", exempted), "name", pod.Name) + observability.PVCUnbinderGateExempted.Inc() + if r.Recorder != nil { + msg := fmt.Sprintf("unbound claims %s are exempted as stuck-Pod claims and the reconciled Pod holds its own mis-pin proof; proceeding past the pvc-rebinding gate", claimListForEvent(exempted)) + r.Recorder.Eventf(&pod, nil, corev1.EventTypeWarning, eventReasonGateExempted, "Exempt", "%s", msg) + } + } } - // Gate 4: a PV we previously freed (ClaimRef cleared under - // --allow-pv-rebinding) is still Available and its node still - // exists — meaning it's a live binding candidate that a NEW PVC - // from a subsequent unbind could mis-pair with (the sequential - // cross-broker swap). Defer all further unbinds for this cluster - // until the freed disk is re-bound or its node is permanently - // gone. See [FreedPVAnnotation]. + // Gate 4 "freed-pv": a PV we freed earlier (--allow-pv-rebinding) + // is still Available and its node still exists, so a new claim + // could bind to the wrong disk. Wait until the disk re-binds or + // its node is gone. See [FreedPVAnnotation]. if pvGates.freedPVUnresolved { const msg = "a previously freed PV is still Available with a live node; deferring to avoid cross-broker rebinding" logger.Info(msg, "name", pod.Name) @@ -676,54 +818,35 @@ type pvGateState struct { freedPVUnresolved bool } -// checkPVGates evaluates Gates 0 and 4 in a single uncached pass over -// the PV list. All reads go through the uncached Reader: these -// annotations are written by this controller moments before they're -// needed (exactly where the informer cache lags), and being durable -// API-server state is what makes the gates survive restarts and -// leader handoffs. +// checkPVGates evaluates Gates 0 and 4 in one uncached pass over the +// PV list. // -// Per PV carrying [InFlightAnnotation] == clusterKey (Gate 0): the -// recorded claim is fetched (uncached). If it's missing, still shows -// the recorded (old) UID, is Terminating, or is unbound — the -// previous unbind hasn't settled and unbindInFlight is set. Once the -// claim is observed recreated (new UID) and bound, the in-flight -// annotations are cleared. +// Gate 0: for each PV with [InFlightAnnotation] == clusterKey, fetch +// its recorded claim. The unbind has settled — and the annotations +// are cleared — once the claim is either recreated (new UID) and +// bound, or still carries the old UID and is not Terminating (the +// delete never happened; pre-unbind state, safe to retry from). A +// missing, Terminating, or recreated-but-unbound claim keeps +// unbindInFlight set. // -// Per PV carrying [FreedPVAnnotation] == clusterKey (Gate 4): +// Gate 4: for each PV with [FreedPVAnnotation] == clusterKey: +// Bound again → clear the annotation. Available with its pinned node +// still existing → a live rebinding candidate; freedPVUnresolved is +// set. Available with the node gone → inert; not blocking, but the +// annotation is kept in case the node name is reused. // -// - Bound again → the freed disk found a claim (ideally its original -// broker's recreated PVC). Clear the annotation. -// - Available + pinned node EXISTS → live rebinding candidate; a new -// PVC from another unbind could mis-pair with it. -// freedPVUnresolved is set. -// - Available + pinned node GONE (Node object deleted) → inert: with -// WaitForFirstConsumer, no pod can ever schedule onto a -// nonexistent node, so the binder will never match this PV. Not -// blocking, but the annotation is KEPT — if a node with the same -// name rejoins (name reuse is real with LocalPathProvisioner), the -// PV becomes a live candidate again and the gate re-engages. +// The reconciled Pod's OWN in-flight claims do not block: this +// reconcile is exactly the retry that finishes a stuck unbind (the +// pod delete is what releases a claim held by the pvc-protection +// finalizer). Counting them would deadlock. Siblings still defer. // -// In-flight entries whose recorded claim belongs to the Pod being -// reconciled are NOT counted as blocking. The reconcile for that pod -// is exactly the retry that completes a stuck unbind — most -// importantly the pod-delete step, which is what releases a claim -// held in Terminating by the pvc-protection finalizer. Counting the -// pod's own claims would deadlock: the claim can't finish deleting -// until the pod is deleted, and the pod would never be deleted -// because the gate defers on the claim. Sibling pods still defer. +// If a freed PV never re-binds, or an in-flight claim is never +// recreated, these gates hold the cluster forever. That is on +// purpose: an alertable halt (metric + Event) is better than a silent +// disk swap. Operators fix it by removing the orphaned PV or the +// annotation. // -// If a freed PV never re-binds and its node persists — or an -// in-flight claim is never recreated (e.g. the cluster was scaled -// down mid-unbind) — these gates hold the cluster's unbinds -// indefinitely. That's deliberate: the failure mode is an alertable -// halt (gate metric + Event) instead of a silent cross-broker disk -// swap. Operators resolve it by removing the orphaned PV or, if the -// state is known-good, the annotation itself. -// -// clusterKey == "" (pod without the instance label) short-circuits to -// "no gates engaged" — such pods were never covered by per-cluster -// serialization. +// clusterKey == "" (pod without the instance label) engages no gates. func (r *Controller) checkPVGates(ctx context.Context, clusterKey string, pod *corev1.Pod) (pvGateState, error) { var state pvGateState if clusterKey == "" { @@ -798,25 +921,19 @@ func (r *Controller) inFlightClaimOwnedBy(pv *corev1.PersistentVolume, ownClaims } // inFlightClaimSettled reports whether the claim recorded in a PV's -// [InFlightClaimAnnotation] no longer represents an unbind in -// progress, observed through the uncached Reader. Two states settle: +// [InFlightClaimAnnotation] is done unbinding (uncached read). Two +// states count as settled: // -// - The claim was recreated with a NEW UID and is bound — the -// unbind completed end-to-end. -// - The claim still has the OLD UID and is NOT Terminating — the -// previous reconcile failed between annotating and deleting, so -// no destructive action ever happened. The world is in its -// pre-unbind state, which is safe to proceed (and retry) from. -// Without this case, a failed PVC delete after a successful -// annotation write would deadlock the cluster's unbinder: Gate 0 -// would defer every reconcile, including the retry that would -// re-attempt the delete. +// - the claim was recreated (new UID) and is bound — the unbind +// completed; or +// - the claim still has the OLD UID and is not Terminating — the +// previous reconcile failed before it deleted anything, so the +// world is still in its pre-unbind state and safe to retry from. +// Without this case a failed delete would deadlock Gate 0. // -// Everything else — claim missing (deleted, awaiting StatefulSet -// recreation), Terminating (deletion held by the pvc-protection -// finalizer until the pod is deleted), or recreated-but-unbound — is -// an unbind in progress. A malformed annotation is treated as -// not-settled (conservative) and logged. +// Everything else (claim missing, Terminating, or recreated but not +// bound) is an unbind in progress. A malformed annotation counts as +// not settled. func (r *Controller) inFlightClaimSettled(ctx context.Context, pv *corev1.PersistentVolume) (bool, error) { parts := strings.SplitN(pv.Annotations[InFlightClaimAnnotation], "/", 3) if len(parts) != 3 || parts[0] == "" || parts[1] == "" { @@ -883,22 +1000,41 @@ func (r *Controller) freedPVBlocking(ctx context.Context, pv *corev1.PersistentV return true, nil } - var node corev1.Node - err := r.reader().Get(ctx, client.ObjectKey{Name: hostname}, &node) - switch { - case err == nil: - // Node exists (Ready or not — cordoned/NotReady nodes can - // recover and bind). Live candidate; defer. - return true, nil - case apierrors.IsNotFound(err): + // Resolve the pinned node by the kubernetes.io/hostname LABEL, + // never the Node object name — kubelet --hostname-override makes + // them differ, and a name-based Get would report a live node as + // gone and OPEN this gate. Mirrors the exemption path + // ([Controller.nodeUnavailableForScheduling]). + var nodeList corev1.NodeList + if err := r.reader().List(ctx, &nodeList, client.MatchingLabels{corev1.LabelHostname: hostname}); err != nil { + // Out-of-band RBAC can lag the upgrade that introduced this + // LIST (the lookup it replaced needed only `get`). Degrade + // Forbidden to the conservative answer — a live candidate, so + // the gate stays engaged and defers with its usual paper + // trail — instead of error-looping the reconcile with no + // metric or Event. + if apierrors.IsForbidden(err) { + log.FromContext(ctx).Info("nodes LIST forbidden; treating the freed PV as a live rebinding candidate", "name", pv.Name, "reason", err.Error()) + return true, nil + } + return false, err + } + if len(nodeList.Items) == 0 { // Node permanently gone; PV is inert. Keep annotation in case // of node-name reuse, but don't defer on it. return false, nil - default: - return false, err } + // A node with this hostname exists (Ready or not — cordoned and + // NotReady nodes can recover and bind). Live candidate; defer. + return true, nil } +// ShouldRemediate reports whether a Pod qualifies for remediation: it +// matches the Selector, is a Pending StatefulSet pod, and its +// Unschedulable condition matches the scheduling-failure signature. +// If the condition is younger than Timeout, it returns (true, wait): +// qualified, but re-check after `wait` in case the scheduler settles +// it on its own. func (r *Controller) ShouldRemediate(ctx context.Context, pod *corev1.Pod) (bool, time.Duration) { if r.Selector != nil && !r.Selector.Matches(labels.Set(pod.Labels)) { log.FromContext(ctx).Info("selector not satisfied; skipping", "name", pod.Name, "labels", pod.Labels, "selector", r.Selector.String()) @@ -916,14 +1052,12 @@ func (r *Controller) ShouldRemediate(ctx context.Context, pod *corev1.Pod) (bool cond := pod.Status.Conditions[idx] - // Short of re-implementing or importing scheduler, this is the best way to - // detect if a scheduling failure is _likely_ due to volume node affinity - // conflict. We check for a either an explicit mention of volume node - // affinity issues OR a message indicating that no nodes within the cluster - // may host this Pod. - // As of Kubernetes >1.21.x <=1.28.x (Didn't track down an exact version), - // volume node affinity conflicts no longer seem to appear in the message, - // hence the need to check for a much weaker case. + // The message check is deliberately weak. Schedulers stopped + // naming volume node affinity in the message somewhere between + // K8s 1.21 and 1.28 (exact version never tracked down), so we + // accept either an explicit mention or any "0/N nodes are + // available" total failure. Stronger proof comes later, from the + // exemption evidence chain, not from message text. if !SchedulingFailureRE.MatchString(cond.Message) { log.FromContext(ctx).Info("scheduling failure does not appear to indicate volume affinity issues; skipping", "name", pod.Name, "condition", cond) return false, 0 @@ -936,6 +1070,8 @@ func (r *Controller) ShouldRemediate(ctx context.Context, pod *corev1.Pod) (bool return true, 0 } +// pvcUnbinderPredicate is the cheap event filter: only Pending Pods +// owned by a StatefulSet are interesting to this controller. func pvcUnbinderPredicate(obj client.Object) bool { pod, ok := obj.(*corev1.Pod) if !ok { @@ -951,13 +1087,12 @@ func pvcUnbinderPredicate(obj client.Object) bool { return stsManaged && isPending } -// clusterKey identifies the Redpanda cluster this Pod belongs to; it's -// the value written into the PV gate annotations ([InFlightAnnotation], -// [FreedPVAnnotation]) to scope Gates 0 and 4 per cluster. Returns "" -// if the Pod lacks the standard app.kubernetes.io/instance label (in -// which case the per-cluster gates are skipped — the original unscoped -// behavior). Includes the ClusterName prefix when running under -// MulticlusterController so keys are unique across K8s clusters. +// clusterKey identifies the Redpanda cluster a Pod belongs to. It is +// the value written into the gate annotations, scoping Gates 0 and 4 +// per cluster. Returns "" when the Pod has no +// app.kubernetes.io/instance label; the per-cluster gates are then +// skipped. The ClusterName prefix keeps keys unique in multicluster +// mode. func (r *Controller) clusterKey(pod *corev1.Pod) string { instance := pod.Labels[operatorlabels.InstanceKey] if instance == "" { @@ -966,28 +1101,14 @@ func (r *Controller) clusterKey(pod *corev1.Pod) string { return r.ClusterName + "/" + pod.Namespace + "/" + instance } -// isClusterPaused returns true if any of the Redpanda CR types that could -// own the given Pod carries the PauseAnnotation set to "true". The Pod -// is linked to its CR via the standard app.kubernetes.io/instance label. -// -// Three candidate types are tried in order (matching name+namespace): -// -// - v1alpha2.Redpanda — single-cluster v2 deployments. -// - v1alpha2.StretchCluster — multi-cluster/stretched v2 deployments. -// Broker pods belonging to a StretchCluster member carry the -// StretchCluster's name in the instance label. -// - v1alpha1.Cluster — legacy v1 deployments. -// -// If ANY of these has the pause annotation set, the pod is paused. We -// gracefully ignore three "we can't ask about this type" categories so -// the same code works in every operator binary regardless of which -// types/CRDs are installed: -// -// - apierrors.IsNotFound: the CR doesn't exist in this namespace. -// - meta.IsNoMatchError: the CRD isn't installed on the API server. -// - runtime.IsNotRegisteredError: the Go type isn't in this -// controller's scheme (e.g. multicluster mode, which only has the -// v2 types registered). +// isClusterPaused reports whether the Pod's owning CR carries +// [PauseAnnotation] = "true". The Pod is linked to its CR by the +// app.kubernetes.io/instance label. Three CR types are checked: +// v1alpha2.Redpanda, v1alpha2.StretchCluster (a member's broker pods +// carry the StretchCluster's name in the instance label), and the +// legacy v1alpha1.Cluster. Errors that only mean "this type is not reachable +// here" (CR absent, CRD not installed, type not in scheme) are +// ignored so the same code runs in every operator binary. func (r *Controller) isClusterPaused(ctx context.Context, pod *corev1.Pod) (bool, error) { instance := pod.Labels[operatorlabels.InstanceKey] if instance == "" { @@ -1034,29 +1155,15 @@ func cannotCheckCRType(err error) bool { return apierrors.IsNotFound(err) || meta.IsNoMatchError(err) || runtime.IsNotRegisteredError(err) } -// multiNodeEventInProgress reports whether the set of currently-stuck -// Redpanda broker pods spans more than one distinct node. Returning -// true means the unbinder should defer — the symptom matches a K8s-wide -// event (cloud upgrade, AZ flake, node-pool surge) rather than a -// single-node failure. -// -// Counting distinct nodes rather than distinct pods matters for the -// case where multiple co-tenant pods share a failed node — that's a -// legitimate single-node failure that the unbinder *should* act on, -// not a multi-node K8s event. -// -// The pod list is scoped to Redpanda broker pods (see the -// managedByLabelValue / brokerLabelKey constants for the two-selector -// union and why each is needed) so unrelated workloads with stuck -// local-PV pods can't push this gate to "multi-node" and cause silent -// inaction. Cross-Redpanda-cluster events are still caught because -// every broker pod carries at least one of the two labels. Pods whose -// PV / NodeAffinity / hostname can't be resolved are skipped (we -// can't classify them as same-or-different). -// -// This gate is best-effort by design: it reads cached lists, so a -// fast-moving multi-node event may be under-counted. The binding -// safety invariant rests on Gates 0/3/4, not on this gate. +// multiNodeEventInProgress reports whether stuck Redpanda broker pods +// are pinned to more than one distinct node (Gate 2). If so, the +// symptom looks like a cluster-wide event and the unbinder defers. +// Nodes are counted, not pods: several pods on one dead node is a +// single-node failure the unbinder should act on. The pod list is +// limited to broker pods (see the label constants) so unrelated +// workloads cannot trip this gate. Unresolvable PVs are skipped. This +// gate is best-effort by design (cached reads); safety rests on +// Gates 0, 3, and 4. func (r *Controller) multiNodeEventInProgress(ctx context.Context) (bool, error) { var pvList corev1.PersistentVolumeList if err := r.Client.List(ctx, &pvList); err != nil { @@ -1137,22 +1244,12 @@ func (r *Controller) multiNodeEventInProgress(ctx context.Context) (bool, error) return false, nil } -// NodeFromPVAffinity extracts the hostname value pinned by a PV's -// NodeAffinity, used by Gate 2 to bucket stuck pods by their pinned -// node. Only `kubernetes.io/hostname` `In` selectors with a single -// value are recognized — that's the shape Local / HostPath volumes -// use, and the actual unbinder only ever acts on those (see the -// `pv.Spec.HostPath == nil && pv.Spec.Local == nil` filter in -// Reconcile). PVs with zone-topology affinity, NotIn selectors, -// multi-value `In`, or unfamiliar keys aren't in the unbinder's scope -// in the first place, so they don't need to contribute to Gate 2's -// distinct-node count. -// -// Gate 2 is best-effort regardless: any PV we can't classify just -// doesn't contribute to the count. The binding safety invariant rests -// on Gates 0/3/4, not on this function's coverage. -// NodeFromPVAffinity extracts the hostname from a PV's NodeAffinity. -// Returns "" if no single-hostname affinity is found. +// NodeFromPVAffinity returns the single hostname a PV's NodeAffinity +// pins it to, for Gate 2's per-node bucketing. Only the shape that +// Local/HostPath volumes use is recognized: one kubernetes.io/hostname +// `In` selector with one value. Anything else returns "" and simply +// does not contribute to Gate 2's count (Gate 2 is best-effort; the +// exemption chain uses the stricter [pvPinnedHostnames] instead). func NodeFromPVAffinity(pv *corev1.PersistentVolume) string { if pv.Spec.NodeAffinity == nil || pv.Spec.NodeAffinity.Required == nil { return "" @@ -1243,17 +1340,19 @@ func DeadNodePVCs(ctx context.Context, c client.Client, apiReader client.Reader, // listClusterPVCsByName returns a name→PVC snapshot for the PVCs that // belong to the same Redpanda/Cluster as `pod` (matched by the // app.kubernetes.io/instance label). Gate 3 inspects spec.volumeName -// on each entry to detect a PVC that's not yet bound to a PV. +// on each entry to detect a PVC that's not yet bound to a PV. The +// caller picks the reader: cached for the deferral fast path, the +// uncached APIReader when the answer helps grant passage. // // Returns an empty (non-nil) map when the Pod has no instance label. -func (r *Controller) listClusterPVCsByName(ctx context.Context, pod *corev1.Pod) (map[string]corev1.PersistentVolumeClaim, error) { +func (r *Controller) listClusterPVCsByName(ctx context.Context, reader client.Reader, pod *corev1.Pod) (map[string]corev1.PersistentVolumeClaim, error) { out := map[string]corev1.PersistentVolumeClaim{} instance := pod.Labels[operatorlabels.InstanceKey] if instance == "" { return out, nil } var pvcList corev1.PersistentVolumeClaimList - if err := r.Client.List(ctx, &pvcList, &client.ListOptions{ + if err := reader.List(ctx, &pvcList, &client.ListOptions{ Namespace: pod.Namespace, LabelSelector: labels.SelectorFromSet(labels.Set{ operatorlabels.InstanceKey: instance, @@ -1267,6 +1366,591 @@ func (r *Controller) listClusterPVCsByName(ctx context.Context, pod *corev1.Pod) return out, nil } +// stuckClaimNames computes Gate 3's exemption set: the names of +// claims that are unbound BECAUSE their Pod is provably deadlocked, +// so waiting on them would wait forever. It also returns whether the +// reconciled Pod itself holds a mis-pinned bound claim (the caller +// re-uses that as the own-proof check before destruction). +// +// Why exempt at all: under WaitForFirstConsumer a stuck Pod's claim +// binds only after the Pod schedules, and the Pod schedules only +// after the unbinder frees its mis-pinned claim — the very action +// Gate 3 would defer. The exemption is symmetric across victims so +// two mis-pinned brokers do not defer on each other; Gate 0 still +// serializes the destructive work. +// +// No Pod is exempted for free. The reconciled Pod and every sibling +// must prove the full deadlock shape via +// [Controller.exemptClaimNames]; a Pod that only matches the weak +// `schedulingFailureRE` message (it may be stuck on CPU, quota, or a +// provisioner failure) proves nothing. A sibling must also pass +// [Controller.ShouldRemediate] in full — same Selector, same +// predicate, and the same r.Timeout freshness check — because a +// sibling that turned Pending only seconds ago may still resolve on +// its own, and Gate 0 has no annotation yet to backstop acting early +// on its behalf. +// +// All reads here are uncached. A lagging informer could keep showing +// a sibling as stuck after it was actually recreated or scheduled, +// and that stale view must not re-create an exemption for a claim +// that is now genuinely settling. +// +// Threat note: any principal with pods/create in this namespace can +// forge a "stuck sibling" (ownerReferences, volumes, affinity, and an +// impossible resource request are all under its control). The mis-pin +// proof itself is also partly self-supplied — the tolerations and +// anti-affinity terms it consults come from the pod's own spec — so +// the evidence chain must never be treated as tamper-resistant +// against such a principal. What actually bounds the damage is scope +// confinement, not the proof: the pipeline deletes only the +// RECONCILED Pod's own claims, and only after that Pod proves its own +// mis-pin. +// +// Claims with no Pod at all (for example, orphaned by an aborted +// scale-up) always defer. Names need no namespace: every list here is +// scoped to pod.Namespace. +func (r *Controller) stuckClaimNames(ctx context.Context, pod *corev1.Pod, unbound []string) (map[string]struct{}, bool, error) { + // The reconciled pod's mis-pin proof is computed exactly once and + // returned to the caller, which needs it again after the Gate 3 + // loop (the own-proof check before destruction). + podMispinned, err := r.podHasMispinnedBoundClaim(ctx, pod) + if err != nil { + return nil, false, err + } + out := map[string]struct{}{} + if podMispinned { + own, err := r.unboundWFFCClaimNames(ctx, pod) + if err != nil { + return nil, false, err + } + for name := range own { + out[name] = struct{}{} + } + } + instance := pod.Labels[operatorlabels.InstanceKey] + if instance == "" { + return out, podMispinned, nil + } + var podList corev1.PodList + if err := r.reader().List(ctx, &podList, &client.ListOptions{ + Namespace: pod.Namespace, + LabelSelector: labels.SelectorFromSet(labels.Set{ + operatorlabels.InstanceKey: instance, + }), + }); err != nil { + return nil, false, err + } + for i := range podList.Items { + p := &podList.Items[i] + if p.Name == pod.Name { + // Already evaluated above; re-running the evidence chain + // would double the live API-server reads for nothing. + continue + } + // A sibling can only exempt its own claims, so a sibling that + // owns none of the unbound claims cannot change the outcome. + // Skipping it avoids a full evidence run (PVC Gets, node + // LISTs, occupant LISTs) per stuck-but-irrelevant pod — which + // would otherwise repeat every 30s against the live API + // server during an incident. + if !slices.ContainsFunc(StsPVCs(p), func(key client.ObjectKey) bool { + return slices.Contains(unbound, key.Name) + }) { + continue + } + // Tag the sibling-qualification logs: ShouldRemediate's + // "skipping" lines would otherwise print the sibling's name in + // the reconciled pod's context every 30s and read as + // remediation decisions about the sibling rather than + // exemption-evidence checks. + sibCtx := log.IntoContext(ctx, log.FromContext(ctx).WithValues("phase", "gate3-exemption", "sibling", p.Name)) + if ok, requeueAfter := r.ShouldRemediate(sibCtx, p); !ok || requeueAfter > 0 { + continue + } + exempt, err := r.exemptClaimNames(sibCtx, p) + if err != nil { + return nil, false, err + } + for name := range exempt { + out[name] = struct{}{} + } + } + return out, podMispinned, nil +} + +// exemptClaimNames returns the pod's own claims that qualify for the +// Gate 3 exemption. Two conditions, both required: +// +// 1. the pod holds a Bound claim on a HostPath/Local PV whose pinned +// node is provably unavailable ([Controller.podHasMispinnedBoundClaim] +// — the claim that actually causes the deadlock); and +// 2. the returned claims are the pod's unbound claims that use a +// WaitForFirstConsumer StorageClass. A claim unbound under +// Immediate binding signals a provisioning failure, not this +// deadlock, and keeps deferring. +// +// PVC reads are uncached: this evidence opens a gate in front of +// destructive deletion, so it must reflect true API-server state. +// +// Deliberately NOT required: that the Pod's Pending message names +// volume affinity. The mis-pin proof stands on its own — a Bound +// local PV confines the Pod to one node, and if that node is proven +// unavailable the Pod cannot schedule there, whatever the aggregate +// scheduler message blames on other nodes. Kubernetes does not +// reliably name volume affinity in the message (the production +// incident's message never did), so requiring it would re-open the +// exact gap this exemption closes. Freeing a claim that is dead-ended +// on an unavailable node is never harmful; at worst it is not enough +// by itself. +func (r *Controller) exemptClaimNames(ctx context.Context, pod *corev1.Pod) (map[string]struct{}, error) { + mispinned, err := r.podHasMispinnedBoundClaim(ctx, pod) + if err != nil { + return nil, err + } + if !mispinned { + return map[string]struct{}{}, nil + } + return r.unboundWFFCClaimNames(ctx, pod) +} + +// unboundWFFCClaimNames returns the names of pod's own StatefulSet +// claims that are unbound AND use a WaitForFirstConsumer StorageClass. +// It is the claim-collection half of [Controller.exemptClaimNames]; +// callers must establish the mis-pin proof first. PVC reads are +// uncached (they feed an exemption decision). +func (r *Controller) unboundWFFCClaimNames(ctx context.Context, pod *corev1.Pod) (map[string]struct{}, error) { + out := map[string]struct{}{} + for _, key := range StsPVCs(pod) { + var pvc corev1.PersistentVolumeClaim + if err := r.reader().Get(ctx, key, &pvc); err != nil { + if apierrors.IsNotFound(err) { + continue + } + return nil, err + } + if pvc.Spec.VolumeName != "" { + continue + } + if wffc, err := r.claimUsesWaitForFirstConsumer(ctx, &pvc); err != nil { + return nil, err + } else if wffc { + out[key.Name] = struct{}{} + } + } + return out, nil +} + +// podHasMispinnedBoundClaim is the mis-pin proof: it reports whether +// the pod holds a Bound claim on a HostPath/Local PV (the only shape +// the unbinder ever acts on) whose EVERY eligible node is unavailable +// to the pod. This is what makes a Pending pod "provably deadlocked" +// instead of merely stuck: nearly every broker holds a bound local +// claim, and the weak scheduling message also fires on CPU or quota +// failures, so the shape alone proves nothing — the node must be +// proven unavailable too. +// +// The PV's NodeAffinity can accept several nodes ([pvPinnedHostnames]). +// If even one of them is available, the pod's failure to schedule +// cannot be blamed on this claim, so it is not proof. A PV whose node +// set cannot be fully resolved is skipped, never guessed at. +// +// "Bound" is proven by the PV's ClaimRef back-reference (namespace, +// name, and UID all matching the claim), never by the claim's +// volumeName alone — that field is user-settable at creation. +// +// The PVC read is uncached: a stale volumeName pointing at an old PV +// would fabricate the evidence. The PV read stays cached because the +// fields used (NodeAffinity, HostPath/Local, ClaimRef UID) never +// change on a live Bound PV. +func (r *Controller) podHasMispinnedBoundClaim(ctx context.Context, pod *corev1.Pod) (bool, error) { + for _, key := range StsPVCs(pod) { + var pvc corev1.PersistentVolumeClaim + if err := r.reader().Get(ctx, key, &pvc); err != nil { + if apierrors.IsNotFound(err) { + continue + } + return false, err + } + if pvc.Spec.VolumeName == "" { + continue + } + var pv corev1.PersistentVolume + if err := r.Client.Get(ctx, client.ObjectKey{Name: pvc.Spec.VolumeName}, &pv); err != nil { + if apierrors.IsNotFound(err) { + continue + } + return false, err + } + // The reference must be a real two-way binding. volumeName is + // user-settable at claim creation (static pre-binding), so on + // its own it proves nothing: a claim pre-pointed at an + // arbitrary local PV must not mint mis-pin evidence. Only the + // binder completes the back-reference with the claim's UID. + // (The destructive pipeline filters PVs by ClaimRef + // namespace/name only; the UID match here is deliberately + // stricter, and Gate 0's settle check self-heals any + // stale-UID PV it stamps.) + if pv.Spec.ClaimRef == nil || + pv.Spec.ClaimRef.Namespace != pvc.Namespace || + pv.Spec.ClaimRef.Name != pvc.Name || + pv.Spec.ClaimRef.UID != pvc.UID { + continue + } + if pv.Spec.NodeAffinity == nil || (pv.Spec.HostPath == nil && pv.Spec.Local == nil) { + continue + } + hostnames, ok := pvPinnedHostnames(&pv) + if !ok { + continue + } + allUnavailable := true + for _, hostname := range hostnames { + unavailable, err := r.nodeUnavailableForScheduling(ctx, hostname, pod) + if err != nil { + return false, err + } + if !unavailable { + allUnavailable = false + break + } + } + if allUnavailable { + return true, nil + } + } + return false, nil +} + +// pvPinnedHostnames returns every hostname the PV's Required +// NodeAffinity accepts (terms are OR'd, so their hostname values are +// unioned), plus ok=false when the set cannot be trusted as complete. +// +// ok is false when there is no Required NodeAffinity, or when any +// term is more complex than exactly one "kubernetes.io/hostname In +// [values]" expression (extra expressions, MatchFields, other keys or +// operators). A partial answer would understate where the PV can +// bind, so the caller must treat ok=false as "cannot evaluate", never +// as "no eligible nodes". +// +// This is the strict counterpart of [nodeFromPVAffinity]: that one +// serves best-effort gates; this one backs a deletion decision, where +// collapsing several eligible nodes into one could delete a claim +// that would still bind fine elsewhere. +func pvPinnedHostnames(pv *corev1.PersistentVolume) ([]string, bool) { + if pv.Spec.NodeAffinity == nil || pv.Spec.NodeAffinity.Required == nil { + return nil, false + } + terms := pv.Spec.NodeAffinity.Required.NodeSelectorTerms + if len(terms) == 0 { + return nil, false + } + var hostnames []string + for _, term := range terms { + if len(term.MatchFields) > 0 || len(term.MatchExpressions) != 1 { + return nil, false + } + expr := term.MatchExpressions[0] + if expr.Key != corev1.LabelHostname || expr.Operator != corev1.NodeSelectorOpIn || len(expr.Values) == 0 { + return nil, false + } + hostnames = append(hostnames, expr.Values...) + } + return hostnames, true +} + +// taintNodeNotReady and taintNodeUnreachable are the taints the node +// lifecycle controller puts on a Node whose Ready condition goes +// False (not-ready) or Unknown (unreachable). +const ( + taintNodeNotReady = corev1.TaintNodeNotReady + taintNodeUnreachable = corev1.TaintNodeUnreachable +) + +// nodeUnavailableForScheduling reports whether the node behind +// `hostname` (one of the hostnames a mis-pinned PV accepts) is truly +// unable to host pod — the fact that turns "stuck" into "deadlocked". +// +// The node is found by LISTING Nodes with a matching +// kubernetes.io/hostname label, not by name: NodeAffinity matches the +// label, and the label does not have to equal the object name +// (--hostname-override, manual relabels). Zero matches means the node +// is gone → unavailable. More than one match is a misconfiguration +// this function refuses to interpret → available (fail closed). +// +// A single matching node is unavailable when any of these holds: +// +// - it is cordoned (Spec.Unschedulable); +// - its Ready condition is False/Unknown, or it carries the +// not-ready/unreachable taint — in both forms judged through the +// pod's own tolerations (Ready=False maps to the not-ready taint, +// Ready=Unknown to unreachable). An unconditional toleration (no +// TolerationSeconds) suppresses this leg. That is POLICY for +// --broker-pod-node-unavailable-toleration=-1s deployments, whose +// contract says only Node DELETION means permanent loss: the +// scheduler might refuse the node right now (its NoSchedule twin +// taint is not covered by the injected NoExecute tolerations), +// but a transient partition must never justify deleting data. Do +// not "fix" this to match raw scheduler semantics. Grace-period +// tolerations (finite TolerationSeconds, auto-injected on every +// pod) do NOT suppress the leg — they describe eviction timing, +// not node health; +// - a live pod already on the node matches one of pod's own +// REQUIRED anti-affinity terms ([podRequiredAntiAffinityMatches]) +// — the production-incident shape, where a broker's PV landed on +// a node another broker occupies. Candidate occupants are every +// pod in the pod's own namespace (the only scope interpretable +// terms can name); the term's own LabelSelector decides which of +// them conflict. Occupancy alone proves nothing (soft or custom +// anti-affinity allows co-location), and Terminating or +// Succeeded/Failed pods do not count as occupants. +// +// If none of these hold, the node looks schedulable, so the pod's +// Pending state cannot be blamed on this claim (more likely CPU, +// quota, or unrelated taints) and this returns false. +// +// Every read here is uncached. This evidence directly unlocks +// destructive deletion, and a stale occupant or node view could +// manufacture proof of a conflict that no longer exists. Gate 0 does +// not backstop that: it only tracks the unbinder's OWN past actions. +func (r *Controller) nodeUnavailableForScheduling(ctx context.Context, hostname string, pod *corev1.Pod) (bool, error) { + var nodeList corev1.NodeList + if err := r.reader().List(ctx, &nodeList, client.MatchingLabels{corev1.LabelHostname: hostname}); err != nil { + return false, err + } + if len(nodeList.Items) == 0 { + return true, nil + } + if len(nodeList.Items) > 1 { + return false, nil + } + node := nodeList.Items[0] + if node.Spec.Unschedulable { + return true, nil + } + for _, cond := range node.Status.Conditions { + if cond.Type != corev1.NodeReady || cond.Status == corev1.ConditionTrue { + continue + } + // A not-True Ready condition is judged through the same + // toleration lens as the taint it maps to (Ready=False maps to + // the not-ready taint, Ready=Unknown to the unreachable + // taint). This is a POLICY choice, not scheduler emulation: + // the scheduler may in fact refuse to place the Pod on this + // node right now (the NoSchedule twin taint is not covered by + // the NoExecute-shaped tolerations that + // --broker-pod-node-unavailable-toleration=-1s injects). But + // that flag's contract says only Node-object DELETION signals + // permanent loss, so for such Pods a transiently unreachable + // node must never count as proof that justifies deleting + // data. Do not "fix" this to match raw scheduler semantics — + // that would convert transient partitions into PVC deletion + // for exactly the deployments that opted out of it. + key := taintNodeNotReady + if cond.Status == corev1.ConditionUnknown { + key = taintNodeUnreachable + } + if !podUnconditionallyTolerates(pod.Spec.Tolerations, &corev1.Taint{Key: key, Effect: corev1.TaintEffectNoExecute}) { + return true, nil + } + } + for i := range node.Spec.Taints { + taint := &node.Spec.Taints[i] + if taint.Key != taintNodeNotReady && taint.Key != taintNodeUnreachable { + continue + } + // Judge through the canonical NoExecute lens regardless of the + // taint's actual effect: the node lifecycle controller applies + // these keys with BOTH NoExecute (eviction pass) and NoSchedule + // (condition pass) effects on every NotReady/unreachable node, + // while --broker-pod-node-unavailable-toleration injects + // NoExecute-shaped tolerations only. Checking the raw NoSchedule + // twin against those tolerations would fail the effect match + // and mark the node unavailable on every real NotReady node — + // silently defeating the tolerate-forever carve-out the + // condition leg above implements. Both twins are applied and + // removed together off the same Ready condition, so one lens + // decides for the pair; and an untolerated NoExecute-lens check + // still catches every pod the taints genuinely exclude. + if !podUnconditionallyTolerates(pod.Spec.Tolerations, &corev1.Taint{Key: taint.Key, Effect: corev1.TaintEffectNoExecute}) { + return true, nil + } + } + if pod.Spec.Affinity == nil || pod.Spec.Affinity.PodAntiAffinity == nil || + len(pod.Spec.Affinity.PodAntiAffinity.RequiredDuringSchedulingIgnoredDuringExecution) == 0 { + // No hard anti-affinity at all (nil affinity, soft-only + // podAntiAffinity.type, or a custom/overridden affinity with + // no required terms) — occupancy can't be evaluated as proof, + // so skip the Pod LIST entirely. + return false, nil + } + // Candidates are ALL pods in the pod's namespace, not just + // same-instance ones. Interpretable terms are already restricted + // to own-namespace scope, and the term's own LabelSelector decides + // who conflicts — the scheduler rejects the node for a matching + // occupant from ANY workload, so an instance-scoped list would + // hide such occupants and silently withhold this proof leg for + // custom terms that select beyond the release. + var podList corev1.PodList + if err := r.reader().List(ctx, &podList, &client.ListOptions{Namespace: pod.Namespace}); err != nil { + return false, err + } + for i := range podList.Items { + other := &podList.Items[i] + if other.Name == pod.Name { + continue + } + if other.DeletionTimestamp != nil { + continue + } + if other.Status.Phase == corev1.PodSucceeded || other.Status.Phase == corev1.PodFailed { + continue + } + if other.Spec.NodeName != node.Name { + continue + } + if podRequiredAntiAffinityMatches(pod, other) { + return true, nil + } + } + return false, nil +} + +// hostnameTopologyKey is the per-node topology key used by the +// redpanda chart's default hard anti-affinity. It is the only +// TopologyKey [podRequiredAntiAffinityMatches] accepts, because at +// node granularity "same domain" can be decided without reading node +// labels. +const hostnameTopologyKey = corev1.LabelHostname + +// podRequiredAntiAffinityMatches reports whether one of pod's +// REQUIRED anti-affinity terms matches occupant, proving the shared +// node is off-limits for pod. Real PodAffinityTerm semantics are +// richer than a label match, so a term only counts when its full +// shape is one this function can interpret: +// +// - TopologyKey is exactly [hostnameTopologyKey]. Any other key +// would need node-label lookups to compare topology domains. +// - NamespaceSelector is nil, and Namespaces is empty or names +// exactly pod's own namespace. Both mean "this pod's namespace", +// which is all the caller's namespace-scoped list can verify. +// The explicit single-namespace form matters: the v1 Cluster's +// default hard anti-affinity always sets it. +// - MatchLabelKeys and MismatchLabelKeys are empty; their +// dynamic-selector semantics are not implemented here. +// +// Any other term shape is skipped, never guessed at. Terms are +// judged by SHAPE, not by which chart option produced them: a +// statefulset.podAntiAffinity type "custom" term that happens to be +// hostname-scoped, own-namespace, and matchLabelKeys-free qualifies +// like the default "hard" one; soft (Preferred-only) anti-affinity +// yields no required terms and never qualifies. Skipping only makes +// Gate 3 keep deferring (alertable via the gate metric); it can never +// falsely open the gate. An invalid LabelSelector is skipped the same +// way. +func podRequiredAntiAffinityMatches(pod, occupant *corev1.Pod) bool { + for _, term := range pod.Spec.Affinity.PodAntiAffinity.RequiredDuringSchedulingIgnoredDuringExecution { + if term.TopologyKey != hostnameTopologyKey { + continue + } + if term.NamespaceSelector != nil { + continue + } + if len(term.Namespaces) > 1 || (len(term.Namespaces) == 1 && term.Namespaces[0] != pod.Namespace) { + continue + } + if len(term.MatchLabelKeys) > 0 || len(term.MismatchLabelKeys) > 0 { + continue + } + selector, err := metav1.LabelSelectorAsSelector(term.LabelSelector) + if err != nil { + continue + } + if selector.Matches(labels.Set(occupant.Labels)) { + return true + } + } + return false +} + +// podUnconditionallyTolerates reports whether one of tolerations +// matches taint (per the standard Kubernetes toleration-match rules: +// empty Key/Effect act as wildcards, Operator Exists ignores Value, +// Operator Equal/"" requires it) AND carries no TolerationSeconds — +// i.e. the Pod tolerates the taint indefinitely, not just for a grace +// period before eviction. See [Controller.nodeUnavailableForScheduling] +// for why the grace-period form doesn't count. +func podUnconditionallyTolerates(tolerations []corev1.Toleration, taint *corev1.Taint) bool { + for i := range tolerations { + t := &tolerations[i] + if t.TolerationSeconds != nil { + continue + } + if t.Key != "" && t.Key != taint.Key { + continue + } + if t.Effect != "" && t.Effect != taint.Effect { + continue + } + switch t.Operator { + case corev1.TolerationOpExists: + return true + case corev1.TolerationOpEqual, "": + if t.Value == taint.Value { + return true + } + } + } + return false +} + +// claimUsesWaitForFirstConsumer reports whether pvc binds under a +// WaitForFirstConsumer StorageClass — the only mode in which a claim +// is EXPECTED to sit unbound while its Pod has not scheduled. +// +// The class is resolved exactly the way Kubernetes resolves it +// (mirrors component-helpers' GetPersistentVolumeClaimClass as of +// k8s.io/api v0.35.1; component-helpers is not a dependency, keep the +// copy in sync by hand): the legacy [corev1.BetaStorageClassAnnotation] +// wins whenever the KEY is present, even with an empty value; only +// when the key is absent does Spec.StorageClassName apply. This looks +// backwards but is what the PV controller does, so both fields on one +// claim must resolve the same way here. +// +// There is deliberately no fallback to the cluster's current default +// StorageClass. Defaulting happens once, at admission, by writing +// Spec.StorageClassName onto the object. A claim that still has nil +// there was never defaulted; guessing today's default could disagree +// with what was true at creation. A claim with no class binds only by +// static PV matching, immediately — never via WaitForFirstConsumer — +// so "no class" resolves to false. A named class that does not exist +// also resolves to false (unknown defers). +// +// The read is uncached: VolumeBindingMode only "changes" through +// delete-and-recreate under the same name, which is exactly what a +// lagging informer would hide. An uncached Get also keeps the RBAC +// grant at bare `get`. +func (r *Controller) claimUsesWaitForFirstConsumer(ctx context.Context, pvc *corev1.PersistentVolumeClaim) (bool, error) { + var name string + if class, found := pvc.Annotations[corev1.BetaStorageClassAnnotation]; found { + name = class + } else if pvc.Spec.StorageClassName != nil { + name = *pvc.Spec.StorageClassName + } else { + return false, nil + } + if name == "" { + return false, nil + } + var sc storagev1.StorageClass + if err := r.reader().Get(ctx, client.ObjectKey{Name: name}, &sc); err != nil { + if apierrors.IsNotFound(err) { + return false, nil + } + return false, err + } + return sc.VolumeBindingMode != nil && *sc.VolumeBindingMode == storagev1.VolumeBindingWaitForFirstConsumer, nil +} + // PodHasVolumeAffinityUnschedulable reports whether a Pod is Pending // because the scheduler couldn't satisfy volume node affinity. Used by // [Controller.multiNodeEventInProgress] and by the Broker controller to diff --git a/operator/internal/controller/pvcunbinder/pvcunbinder_internal_test.go b/operator/internal/controller/pvcunbinder/pvcunbinder_internal_test.go index 54d9bf031..be02d7463 100644 --- a/operator/internal/controller/pvcunbinder/pvcunbinder_internal_test.go +++ b/operator/internal/controller/pvcunbinder/pvcunbinder_internal_test.go @@ -13,20 +13,28 @@ import ( "context" "fmt" "testing" + "time" + promtestutil "github.com/prometheus/client_golang/prometheus/testutil" "github.com/stretchr/testify/require" corev1 "k8s.io/api/core/v1" + storagev1 "k8s.io/api/storage/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/tools/events" "k8s.io/utils/ptr" + ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" redpandav1alpha2 "github.com/redpanda-data/redpanda-operator/operator/api/redpanda/v1alpha2" vectorizedv1alpha1 "github.com/redpanda-data/redpanda-operator/operator/api/vectorized/v1alpha1" + "github.com/redpanda-data/redpanda-operator/operator/internal/observability" operatorlabels "github.com/redpanda-data/redpanda-operator/operator/pkg/labels" ) @@ -34,6 +42,7 @@ func newScheme(t *testing.T, withV2, withStretch, withV1 bool) *runtime.Scheme { t.Helper() s := runtime.NewScheme() require.NoError(t, corev1.AddToScheme(s)) + require.NoError(t, storagev1.AddToScheme(s)) if withV2 || withStretch { require.NoError(t, redpandav1alpha2.Install(s)) } @@ -129,8 +138,16 @@ func pvWithAnnotations(name string, phase corev1.PersistentVolumePhase, hostname return pv } +// newNode builds a Node whose kubernetes.io/hostname label matches +// name — the common-case default (kubelet sets it that way absent a +// --hostname-override), and what nodeUnavailableForScheduling actually +// resolves by. Tests exercising a diverging hostname label build their +// own Node object instead. func newNode(name string) *corev1.Node { - return &corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: name}} + return &corev1.Node{ObjectMeta: metav1.ObjectMeta{ + Name: name, + Labels: map[string]string{corev1.LabelHostname: name}, + }} } // TestCheckPVGates exercises the durable, uncached PV-annotation gates @@ -286,6 +303,21 @@ func TestCheckPVGates(t *testing.T) { require.True(t, state.freedPVUnresolved) }) + t.Run("freed: hostname label diverging from the Node name still holds the gate", func(t *testing.T) { + // PV node affinity carries the kubernetes.io/hostname LABEL + // value; under kubelet --hostname-override it differs from the + // Node object's name. Resolving by name would report a live + // node as gone and OPEN this gate — the fail direction that + // re-enables cross-broker rebinding. + pv := pvWithAnnotations("pv-0", corev1.VolumeAvailable, "worker-a", map[string]string{FreedPVAnnotation: key}) + node := newNode("ip-10-0-0-1") + node.Labels[corev1.LabelHostname] = "worker-a" + r := newController(t, s, pv, node) + state, err := r.checkPVGates(ctx, key, otherPod()) + require.NoError(t, err) + require.True(t, state.freedPVUnresolved, "the pinned node must be resolved by its hostname label, not its Node name") + }) + t.Run("freed: Available with node permanently gone does not hold, keeps annotation", func(t *testing.T) { pv := pvWithAnnotations("pv-0", corev1.VolumeAvailable, "node-a", map[string]string{FreedPVAnnotation: key}) r := newController(t, s, pv) // no Node object @@ -331,6 +363,1699 @@ func TestCheckPVGates(t *testing.T) { }) } +// TestReconcileGate3StuckClaimExemption drives Reconcile end-to-end +// through Gate 3 (pvc-rebinding). Gate 3 defers while any claim in the +// cluster is unbound — but claims owned by Pods that are themselves +// stuck Pending on volume affinity (the reconciled Pod first among +// them) can never settle on their own and must be exempt, or the gate +// deadlocks the exact unbind meant to fix them. +// +// The canonical deadlock (from a production incident): a fresh +// cluster's broker never schedules because one of its two claims +// (shadow-index-cache) was provisioned onto an already-occupied node. +// With WaitForFirstConsumer binding, its OTHER claim (datadir) can +// never bind until the Pod schedules, the Pod can never schedule until +// the mis-pinned claim is unbound, and Gate 3 defers that unbind +// forever because the datadir claim has no volumeName. Three-way +// circular wait; the cluster never bootstraps. The exemption must +// hold symmetrically across multiple victims: two mis-pinned brokers +// each holding an unbound datadir claim must not defer on each +// other's. +func TestReconcileGate3StuckClaimExemption(t *testing.T) { + ctx := context.Background() + s := newScheme(t, false, false, false) + + // wffc is a WaitForFirstConsumer StorageClass named "standard". + // PVCs meant to model the deadlock's unbound WFFC claim set + // Spec.StorageClassName to it explicitly, matching what Kubernetes + // itself would have already persisted onto the object at CREATE + // time (via the DefaultStorageClass admission controller) — this + // evaluator deliberately does NOT re-resolve "the cluster's current + // default" for a claim whose Spec.StorageClassName is nil, so nil + // fixtures below are exercising "no class", not "the default". + wffc := &storagev1.StorageClass{ + ObjectMeta: metav1.ObjectMeta{Name: "standard"}, + VolumeBindingMode: ptr.To(storagev1.VolumeBindingWaitForFirstConsumer), + } + + // boundHostPathPV builds a PV that qualifies for unbinding: + // Bound, ClaimRef set, NodeAffinity pinned, HostPath-backed. + boundHostPathPV := func(name, claimName, hostname string) *corev1.PersistentVolume { + pv := newPVWithAffinity(name, "ns", claimName, hostname) + pv.Spec.PersistentVolumeSource = corev1.PersistentVolumeSource{ + HostPath: &corev1.HostPathVolumeSource{Path: "/data"}, + } + pv.Status.Phase = corev1.VolumeBound + return pv + } + + // hardAntiAffinity builds a RequiredDuringSchedulingIgnoredDuringExecution + // PodAntiAffinity term matching matchLabels — the shape the + // redpanda chart renders for the default podAntiAffinity.type: + // hard. Tests needing podAntiAffinity.type: soft/custom instead use + // PreferredDuringScheduling (or leave Affinity nil). + hardAntiAffinity := func(matchLabels map[string]string) *corev1.Affinity { + return &corev1.Affinity{ + PodAntiAffinity: &corev1.PodAntiAffinity{ + RequiredDuringSchedulingIgnoredDuringExecution: []corev1.PodAffinityTerm{{ + LabelSelector: &metav1.LabelSelector{MatchLabels: matchLabels}, + TopologyKey: "kubernetes.io/hostname", + }}, + }, + } + } + + // softAntiAffinity builds a PreferredDuringScheduling-only + // PodAntiAffinity — the podAntiAffinity.type: soft shape, which + // doesn't forbid co-location and so must never count as occupancy + // proof. + softAntiAffinity := func(matchLabels map[string]string) *corev1.Affinity { + return &corev1.Affinity{ + PodAntiAffinity: &corev1.PodAntiAffinity{ + PreferredDuringSchedulingIgnoredDuringExecution: []corev1.WeightedPodAffinityTerm{{ + Weight: 100, + PodAffinityTerm: corev1.PodAffinityTerm{ + LabelSelector: &metav1.LabelSelector{MatchLabels: matchLabels}, + TopologyKey: "kubernetes.io/hostname", + }, + }}, + }, + } + } + + // stuckBroker models the production victim: a Pending pod with a + // volume-affinity scheduling failure holding an unbound WFFC + // datadir claim and a Bound shadow-index-cache claim whose PV is + // pinned to a node the pod can never fit on. + stuckBroker := func(name string) (*corev1.Pod, *corev1.PersistentVolumeClaim, *corev1.PersistentVolumeClaim, *corev1.PersistentVolume) { + pod := withPVC(withPVC(podWithVolumeAffinityFailure(name, "ns", "redpanda"), "datadir-"+name), "shadow-index-cache-"+name) + datadir := newPVC("datadir-"+name, "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") // already defaulted, per real admission-time behavior + shadow := newPVC("shadow-index-cache-"+name, "ns", "redpanda", "pv-shadow-"+name) + pv := boundHostPathPV("pv-shadow-"+name, "shadow-index-cache-"+name, "node-a") + return pod, datadir, shadow, pv + } + + t.Run("own unbound WFFC claim does not deadlock the unbind", func(t *testing.T) { + pod, datadir, shadow, pv := stuckBroker("rp-1") + recorder := &events.FakeRecorder{Events: make(chan string, 8)} + r := newController(t, s, wffc, pod, datadir, shadow, pv) + r.Recorder = recorder + + exemptedBefore := promtestutil.ToFloat64(observability.PVCUnbinderGateExempted) + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "Gate 3 must not defer on the reconciled pod's own unbound claim") + require.Equal(t, exemptedBefore+1, promtestutil.ToFloat64(observability.PVCUnbinderGateExempted), "an exemption-based gate pass must increment the exempted metric") + + // The gate override left its paper trail: an Event naming the + // exempted claim (mirroring the deferral-event assertion + // elsewhere in this suite). + select { + case ev := <-recorder.Events: + require.Contains(t, ev, eventReasonGateExempted) + require.Contains(t, ev, "datadir-rp-1") + default: + t.Fatal("expected a PVCUnbinderGateExempted event when the gate is passed via the exemption") + } + + // The mis-pinned Bound claim was deleted... + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the mis-pinned bound claim must be deleted") + + // ...the unbound claim was left untouched... + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "datadir-rp-1"}, &gotPVC)) + + // ...the PV was prepared for unbind (Retain + in-flight)... + var gotPV corev1.PersistentVolume + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Name: "pv-shadow-rp-1"}, &gotPV)) + require.Equal(t, corev1.PersistentVolumeReclaimRetain, gotPV.Spec.PersistentVolumeReclaimPolicy) + require.Equal(t, "/ns/redpanda", gotPV.Annotations[InFlightAnnotation]) + + // ...and the Pod was deleted to re-trigger PVC creation. + var gotPod corev1.Pod + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod) + require.True(t, apierrors.IsNotFound(err), "the pod must be deleted to re-trigger PVC creation and scheduling") + }) + + t.Run("two stuck pods with unbound WFFC claims do not mutually deadlock", func(t *testing.T) { + // The >=2-victim variant: rp-1 and rp-2 are both mis-pinned to + // the same occupied node, each holding an unbound WFFC datadir + // claim. Neither claim can ever settle on its own, so + // Reconcile(rp-1) must not defer on rp-2's unbound claim (and + // vice versa) — otherwise the deadlock survives with two + // victims. Destructive work stays serialized by Gate 0. + pod1, datadir1, shadow1, pv1 := stuckBroker("rp-1") + pod2, datadir2, shadow2, pv2 := stuckBroker("rp-2") + r := newController(t, s, wffc, pod1, datadir1, shadow1, pv1, pod2, datadir2, shadow2, pv2) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod1)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "Gate 3 must not defer on a fellow stuck pod's unbound claim") + + // rp-1 was remediated... + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "rp-1's mis-pinned bound claim must be deleted") + var gotPod corev1.Pod + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod) + require.True(t, apierrors.IsNotFound(err), "rp-1 must be deleted to re-trigger PVC creation") + + // ...and rp-2 was left for its own reconcile. + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-2"}, &gotPVC)) + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-2"}, &gotPod)) + }) + + t.Run("own unrelated unbound claim under Immediate binding does not exempt a healthy bound claim", func(t *testing.T) { + // rp-1 has TWO claims: "datadir" is unbound, but under an + // Immediate-mode StorageClass — a real provisioning failure + // (storage class, quota, capacity), NOT the WaitForFirstConsumer + // deadlock this exemption targets — and "shadow-index-cache" is + // a perfectly healthy Bound HostPath/Local claim that happens to + // satisfy podHasMispinnedBoundClaim's shape test on its own. + // Because "datadir" isn't WaitForFirstConsumer, it must not be + // exempted, so Gate 3 keeps deferring — the healthy claim must + // survive untouched instead of being deleted as a false-positive + // "mis-pin". + immediate := storagev1.VolumeBindingImmediate + immediateSC := &storagev1.StorageClass{ + ObjectMeta: metav1.ObjectMeta{Name: "immediate-sc"}, + VolumeBindingMode: &immediate, + } + pod := withPVC(withPVC(podWithVolumeAffinityFailure("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("immediate-sc") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + r := newController(t, s, wffc, immediateSC, pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "an Immediate-mode unbound claim must not be exempted; Gate 3 must keep deferring") + + // Nothing was touched — in particular the healthy bound claim + // must survive just because it happens to be a HostPath/Local + // claim with NodeAffinity. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("generic scheduling failure with an available pinned node does not exempt the bound claim", func(t *testing.T) { + // rp-1 matches the weak schedulingFailureRE signature ("0/N + // nodes are available: ... Insufficient cpu") which says + // NOTHING about volumes — it's Pending because every node lacks + // CPU, unrelated to storage. It happens to hold both an unbound + // WFFC "datadir" claim AND a Bound HostPath/Local "cache" claim + // pinned to "node-a" — the same shape a genuinely mis-pinned + // broker has. But "node-a" here actually EXISTS, is not + // cordoned, and no other Pod occupies it: nothing proves that + // pinning is why rp-1 can't schedule. Without that proof, Gate 3 + // must keep deferring rather than nuking a claim that has + // nothing to do with the CPU shortage. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + // WFFC class set so the ONLY thing standing between deferral + // and exemption is the evaluator under test — without it, + // claimUsesWaitForFirstConsumer fails the claim unconditionally + // and the test would pass even with the evaluator broken. + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + r := newController(t, s, wffc, newNode("node-a"), pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a claim pinned to a provably available node must not be exempted; Gate 3 must keep deferring") + + // Nothing was touched. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the bound claim on an available node must not be deleted") + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("multi-value hostname NodeAffinity with one available node does not exempt the bound claim", func(t *testing.T) { + // A PV's NodeAffinity is a label selector, not a single Node + // reference: a single `In` expression can list multiple + // hostname values, meaning the claim can bind on ANY of them. + // Here "node-a" is cordoned but "node-b" — also named in the + // same expression — is healthy and available. rp-1 could still + // bind this claim on node-b, so it's not proven mis-pinned; + // exempting it (and deleting a claim that could still bind + // fine) would be premature. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + // WFFC class set so the ONLY thing standing between deferral + // and exemption is the evaluator under test — without it, + // claimUsesWaitForFirstConsumer fails the claim unconditionally + // and the test would pass even with the evaluator broken. + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + healthyPV.Spec.NodeAffinity.Required.NodeSelectorTerms[0].MatchExpressions[0].Values = []string{"node-a", "node-b"} + cordonedA := newNode("node-a") + cordonedA.Spec.Unschedulable = true + availableB := newNode("node-b") + r := newController(t, s, wffc, cordonedA, availableB, pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a claim that could still bind on an available alternate node must not be exempted; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the claim must not be deleted while an alternate eligible node is available") + }) + + t.Run("multi-value hostname NodeAffinity with every eligible node unavailable is still exempted", func(t *testing.T) { + // Same shape as above, but BOTH hostname values the PV's + // NodeAffinity accepts are confirmed unavailable (one cordoned, + // one simply doesn't exist) — the claim genuinely cannot bind + // anywhere, so the exemption must still fire. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + mispinned := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-mispinned-rp-1") + mispinnedPV := boundHostPathPV("pv-mispinned-rp-1", "shadow-index-cache-rp-1", "node-a") + mispinnedPV.Spec.NodeAffinity.Required.NodeSelectorTerms[0].MatchExpressions[0].Values = []string{"node-a", "node-b"} + cordonedA := newNode("node-a") + cordonedA.Spec.Unschedulable = true + // "node-b" is deliberately absent from the fixture — gone entirely. + r := newController(t, s, wffc, cordonedA, pod, datadir, mispinned, mispinnedPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "a claim whose every eligible node is unavailable must still be exempted") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the claim must be deleted once every eligible node is confirmed unavailable") + }) + + t.Run("two Nodes sharing one hostname label fail closed even when both are cordoned", func(t *testing.T) { + // A hostname label value resolving to MORE than one Node is a + // misconfigured state the evaluator refuses to interpret — + // even when every candidate looks unavailable, it + // conservatively reports the node available and withholds the + // exemption, so Gate 3 keeps deferring. + pod, datadir, shadow, pv := stuckBroker("rp-1") + cordonedA := newNode("node-a") + cordonedA.Spec.Unschedulable = true + doppelganger := &corev1.Node{ObjectMeta: metav1.ObjectMeta{ + Name: "node-a-doppelganger", + Labels: map[string]string{corev1.LabelHostname: "node-a"}, + }} + doppelganger.Spec.Unschedulable = true + r := newController(t, s, wffc, cordonedA, doppelganger, pod, datadir, shadow, pv) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "an ambiguous hostname resolution must fail closed; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the bound claim must not be deleted on ambiguous node resolution") + }) + + t.Run("hostname label diverging from the Node's metadata.name still resolves correctly", func(t *testing.T) { + // A PV's NodeAffinity matches against the kubernetes.io/hostname + // LABEL, not the Node object's own metadata.name — they're + // independent fields that merely default to the same value. + // Here the Node actually named "node-a" is registered under a + // DIFFERENT object name ("actual-node-object-name") with only + // its hostname LABEL set to "node-a", and it's healthy + // (uncordoned, Ready, unoccupied). Resolving by Get(name= + // "node-a") — the pre-fix behavior — would find nothing and + // wrongly conclude the node is gone/unavailable, exempting and + // deleting a perfectly healthy claim. Resolving by the label + // (the fix) finds the real, available Node and correctly + // declines to exempt. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + // WFFC class set so the ONLY thing standing between deferral + // and exemption is the evaluator under test — without it, + // claimUsesWaitForFirstConsumer fails the claim unconditionally + // and the test would pass even with the evaluator broken. + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + divergentNode := &corev1.Node{ + ObjectMeta: metav1.ObjectMeta{ + Name: "actual-node-object-name", + Labels: map[string]string{corev1.LabelHostname: "node-a"}, + }, + } + r := newController(t, s, wffc, divergentNode, pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "resolving by the hostname label must find the real, available Node; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy claim must not be deleted just because no Node is NAMED node-a") + }) + + t.Run("Insufficient-cpu message with a cordoned pinned node is still exempted (intentional, not a false positive)", func(t *testing.T) { + // This is the scenario an adversarial review flagged as a + // possible false positive: rp-1's Pending message blames "3 + // Insufficient cpu" — nothing about storage or volume affinity + // — while its mis-pinned "shadow-index-cache" claim happens to + // sit on a cordoned node. That combination is still correctly + // exempted, and deleting the mis-pinned claim is still + // correct, because the message was never what's being trusted + // here: a Bound HostPath/Local claim's NodeAffinity confines + // rp-1 to exactly "node-a" (Kubernetes will not schedule a Pod + // anywhere its Bound volume isn't pinned to), and "node-a" + // being cordoned is independent, mechanical proof rp-1 cannot + // schedule there — full stop, regardless of what the + // aggregate message additionally blames on other nodes. That + // holds even if a cluster-wide CPU shortage also exists; + // freeing the mis-pinned claim can only help or be neutral, + // never actively harmful, and any residual CPU shortage is + // outside this controller's scope either way. See the doc + // comments on exemptClaimNames and nodeUnavailableForScheduling + // for the full reasoning. Requiring the message to explicitly + // name volume/node affinity instead would reintroduce the + // original production bug this whole exemption exists to fix + // (see stuckClaimNames's doc comment) — Kubernetes doesn't + // reliably surface that attribution. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + mispinned := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-mispinned-rp-1") + mispinnedPV := boundHostPathPV("pv-mispinned-rp-1", "shadow-index-cache-rp-1", "node-a") + cordoned := newNode("node-a") + cordoned.Spec.Unschedulable = true + r := newController(t, s, wffc, cordoned, pod, datadir, mispinned, mispinnedPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "a claim pinned to a cordoned node must still be exempted") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the claim pinned to the cordoned node must be deleted") + }) + + t.Run("generic scheduling failure with a hard-anti-affinity-matching occupant on the pinned node is still exempted", func(t *testing.T) { + // The canonical production shape: rp-1 has a required + // (hard) PodAntiAffinity term matching rp-0's labels, and rp-0 + // already occupies "node-a" (Spec.NodeName) — the exact node + // rp-1's cache PV is pinned to. That's the concrete evidence + // the generic message alone can't provide — the exemption must + // still fire. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + pod.Spec.Affinity = hardAntiAffinity(map[string]string{operatorlabels.InstanceKey: "redpanda"}) + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + mispinned := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-mispinned-rp-1") + mispinnedPV := boundHostPathPV("pv-mispinned-rp-1", "shadow-index-cache-rp-1", "node-a") + occupant := newPod("rp-0", "ns", "redpanda") // carries operatorlabels.InstanceKey: "redpanda", matching pod's required term + occupant.Spec.NodeName = "node-a" + r := newController(t, s, wffc, newNode("node-a"), pod, datadir, mispinned, mispinnedPV, occupant) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "a claim pinned to a node occupied by a hard-anti-affinity match must still be exempted") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the claim pinned to the occupied node must be deleted") + }) + + t.Run("occupant outside the instance label set still counts when the pod's own term selects it", func(t *testing.T) { + // A custom hard anti-affinity term may deliberately select a + // DIFFERENT workload sharing the namespace. The scheduler + // rejects the node for any matching occupant, so the occupant + // scan must consider every pod in the namespace — an + // instance-scoped list would hide this occupant and withhold + // the proof leg the term's shape qualifies for. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + pod.Spec.Affinity = hardAntiAffinity(map[string]string{"app": "other-heavy-workload"}) + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + mispinned := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-mispinned-rp-1") + mispinnedPV := boundHostPathPV("pv-mispinned-rp-1", "shadow-index-cache-rp-1", "node-a") + occupant := &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{ + Name: "heavy-0", + Namespace: "ns", + Labels: map[string]string{"app": "other-heavy-workload"}, + }, + Spec: corev1.PodSpec{NodeName: "node-a"}, + Status: corev1.PodStatus{Phase: corev1.PodRunning}, + } + r := newController(t, s, wffc, newNode("node-a"), pod, datadir, mispinned, mispinnedPV, occupant) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "an anti-affinity-matching occupant from another workload must still prove the node unavailable") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the claim pinned to the occupied node must be deleted") + }) + + t.Run("hard-anti-affinity term that does not match the occupant's labels does not exempt the bound claim", func(t *testing.T) { + // rp-1 DOES have a required PodAntiAffinity term, but it + // selects on a label the occupant doesn't carry (e.g. a + // component-scoped selector that only matches rp-1's own + // pool). An occupant on the pinned node that fails to match + // the term proves nothing, regardless of sharing the instance + // label — Gate 3 must keep deferring. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + pod.Spec.Affinity = hardAntiAffinity(map[string]string{"app.kubernetes.io/component": "redpanda-pool-a-statefulset"}) + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + // WFFC class set so the ONLY thing standing between deferral + // and exemption is the evaluator under test — without it, + // claimUsesWaitForFirstConsumer fails the claim unconditionally + // and the test would pass even with the evaluator broken. + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + occupant := newPod("other-pool-0", "ns", "redpanda") // no "app.kubernetes.io/component" label at all + occupant.Spec.NodeName = "node-a" + r := newController(t, s, wffc, newNode("node-a"), pod, datadir, healthy, healthyPV, occupant) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a non-matching occupant must not be treated as proof of unavailability; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + // assertUnsupportedTermShapeDefers builds rp-1 with a single + // RequiredDuringSchedulingIgnoredDuringExecution term (using + // matchLabels that WOULD match rp-0) and an rp-0 occupant sitting + // on rp-1's pinned node, then asserts Gate 3 still defers: real + // Kubernetes PodAffinityTerm semantics are richer than a bare + // LabelSelector match (Namespaces/NamespaceSelector, + // TopologyKey-scoped node-label comparison, + // matchLabelKeys/mismatchLabelKeys), and podRequiredAntiAffinityMatches + // deliberately doesn't implement all of it — so a term outside the + // one shape it does interpret must not be trusted as proof, even + // though a naive selector-only check (the very bug this guards + // against) would have matched. + assertUnsupportedTermShapeDefers := func(t *testing.T, term corev1.PodAffinityTerm) { + t.Helper() + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + pod.Spec.Affinity = &corev1.Affinity{ + PodAntiAffinity: &corev1.PodAntiAffinity{ + RequiredDuringSchedulingIgnoredDuringExecution: []corev1.PodAffinityTerm{term}, + }, + } + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + // WFFC class set so the ONLY thing standing between deferral + // and exemption is the evaluator under test — without it, + // claimUsesWaitForFirstConsumer fails the claim unconditionally + // and the test would pass even with the evaluator broken. + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + occupant := newPod("rp-0", "ns", "redpanda") // carries operatorlabels.InstanceKey: "redpanda" + occupant.Spec.NodeName = "node-a" + r := newController(t, s, wffc, newNode("node-a"), pod, datadir, healthy, healthyPV, occupant) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "an unsupported term shape must not be treated as proof of unavailability; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + } + + t.Run("required term naming a different namespace does not exempt the bound claim", func(t *testing.T) { + // Namespaces naming a namespace OTHER than rp-1's own ("ns") + // changes the term's meaning away from "this pod's own + // namespace" — this evaluator only ever looks at pods in pod's + // own namespace, so it can't correctly decide whether such a + // term is satisfied and skips it rather than misread it. + assertUnsupportedTermShapeDefers(t, corev1.PodAffinityTerm{ + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{operatorlabels.InstanceKey: "redpanda"}}, + TopologyKey: "kubernetes.io/hostname", + Namespaces: []string{"other-ns"}, + }) + }) + + t.Run("required term naming multiple namespaces (including rp-1's own) does not exempt the bound claim", func(t *testing.T) { + // A multi-element Namespaces list — even one that happens to + // include rp-1's own namespace — applies to the UNION of those + // namespaces, which this evaluator can't fully verify (it only + // scans rp-1's own namespace), so it's skipped rather than + // misread as "own namespace only". + assertUnsupportedTermShapeDefers(t, corev1.PodAffinityTerm{ + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{operatorlabels.InstanceKey: "redpanda"}}, + TopologyKey: "kubernetes.io/hostname", + Namespaces: []string{"ns", "other-ns"}, + }) + }) + + t.Run("required term with Namespaces set to exactly the pod's own namespace is still exempted", func(t *testing.T) { + // Per the v1 Cluster's default hard anti-affinity + // (operator/pkg/resources.StatefulSetResource.obj), Namespaces + // is always set explicitly to []string{pod.Namespace} rather + // than left unset. Per the API's own doc, that's semantically + // IDENTICAL to leaving Namespaces unset — both mean "this + // pod's own namespace" — so it must still count as proof, not + // be skipped just because the field happens to be non-empty. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + pod.Spec.Affinity = &corev1.Affinity{ + PodAntiAffinity: &corev1.PodAntiAffinity{ + RequiredDuringSchedulingIgnoredDuringExecution: []corev1.PodAffinityTerm{{ + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{operatorlabels.InstanceKey: "redpanda"}}, + Namespaces: []string{"ns"}, // rp-1's own namespace, named explicitly + TopologyKey: "kubernetes.io/hostname", + }}, + }, + } + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + mispinned := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-mispinned-rp-1") + mispinnedPV := boundHostPathPV("pv-mispinned-rp-1", "shadow-index-cache-rp-1", "node-a") + occupant := newPod("rp-0", "ns", "redpanda") + occupant.Spec.NodeName = "node-a" + r := newController(t, s, wffc, newNode("node-a"), pod, datadir, mispinned, mispinnedPV, occupant) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "an explicit Namespaces naming only the pod's own namespace must still be treated as proof of unavailability") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the claim pinned to the occupied node must be deleted") + }) + + t.Run("required term with a NamespaceSelector does not exempt the bound claim", func(t *testing.T) { + assertUnsupportedTermShapeDefers(t, corev1.PodAffinityTerm{ + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{operatorlabels.InstanceKey: "redpanda"}}, + TopologyKey: "kubernetes.io/hostname", + NamespaceSelector: &metav1.LabelSelector{}, // matches all namespaces per the API's own semantics + }) + }) + + t.Run("required term with a non-hostname TopologyKey does not exempt the bound claim", func(t *testing.T) { + // A zone/region (or any other custom) TopologyKey would + // require comparing the pinned Node's own label value against + // occupant's Node's label value — this evaluator doesn't + // resolve Node labels for that, so it can't safely assume + // same-Node implies same-domain for a key other than hostname. + assertUnsupportedTermShapeDefers(t, corev1.PodAffinityTerm{ + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{operatorlabels.InstanceKey: "redpanda"}}, + TopologyKey: "topology.kubernetes.io/zone", + }) + }) + + t.Run("required term with a missing TopologyKey does not exempt the bound claim", func(t *testing.T) { + // Empty TopologyKey is invalid on a real API object, but + // defensively handled the same way as any other unsupported key. + assertUnsupportedTermShapeDefers(t, corev1.PodAffinityTerm{ + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{operatorlabels.InstanceKey: "redpanda"}}, + TopologyKey: "", + }) + }) + + t.Run("required term with MatchLabelKeys does not exempt the bound claim", func(t *testing.T) { + assertUnsupportedTermShapeDefers(t, corev1.PodAffinityTerm{ + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{operatorlabels.InstanceKey: "redpanda"}}, + TopologyKey: "kubernetes.io/hostname", + MatchLabelKeys: []string{"apps.kubernetes.io/pod-index"}, + }) + }) + + t.Run("required term with MismatchLabelKeys does not exempt the bound claim", func(t *testing.T) { + assertUnsupportedTermShapeDefers(t, corev1.PodAffinityTerm{ + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{operatorlabels.InstanceKey: "redpanda"}}, + TopologyKey: "kubernetes.io/hostname", + MismatchLabelKeys: []string{"apps.kubernetes.io/pod-index"}, + }) + }) + + t.Run("required term with an invalid LabelSelector does not exempt the bound claim", func(t *testing.T) { + // A term whose selector fails LabelSelectorAsSelector is + // skipped rather than treated as an error or a wildcard — + // malformed affinity is the admission layer's problem to + // reject, not a reason to guess at occupancy proof. + assertUnsupportedTermShapeDefers(t, corev1.PodAffinityTerm{ + LabelSelector: &metav1.LabelSelector{MatchExpressions: []metav1.LabelSelectorRequirement{{ + Key: "app", + Operator: "Bogus", + Values: []string{"x"}, + }}}, + TopologyKey: "kubernetes.io/hostname", + }) + }) + + t.Run("soft-only podAntiAffinity with a matching same-instance occupant does not exempt the bound claim", func(t *testing.T) { + // Exactly the regression this exemption must not misfire on: + // the redpanda chart's podAntiAffinity.type can be "soft" + // (PreferredDuringScheduling — doesn't forbid co-location) or + // "custom", and a Pod's template can override affinity + // entirely. Here rp-1 only has a PREFERRED anti-affinity term + // (or, in the "absent" variant, none at all); rp-0 — otherwise + // a perfectly matching occupant on the same node — proves + // nothing under soft/absent affinity, so a generic CPU failure + // must not be escalated into deleting rp-1's healthy bound + // claim. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + pod.Spec.Affinity = softAntiAffinity(map[string]string{operatorlabels.InstanceKey: "redpanda"}) + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + // WFFC class set so the ONLY thing standing between deferral + // and exemption is the evaluator under test — without it, + // claimUsesWaitForFirstConsumer fails the claim unconditionally + // and the test would pass even with the evaluator broken. + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + occupant := newPod("rp-0", "ns", "redpanda") // would match pod's selector if the term were required + occupant.Spec.NodeName = "node-a" + r := newController(t, s, wffc, newNode("node-a"), pod, datadir, healthy, healthyPV, occupant) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "soft anti-affinity must not be treated as proof of unavailability; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + t.Run("absent Affinity with a matching same-instance occupant does not exempt the bound claim", func(t *testing.T) { + // The "absent" counterpart to the soft-only case above: rp-1 + // has no Affinity at all (e.g. podAntiAffinity.type: custom + // with an empty custom block, or a hand-edited pod template + // with no anti-affinity). rp-0 sits on the pinned node and + // would match a same-instance selector, but with nothing to + // prove a scheduling conflict, Gate 3 must keep deferring. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + // WFFC class set so the ONLY thing standing between deferral + // and exemption is the evaluator under test — without it, + // claimUsesWaitForFirstConsumer fails the claim unconditionally + // and the test would pass even with the evaluator broken. + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + occupant := newPod("rp-0", "ns", "redpanda") + occupant.Spec.NodeName = "node-a" + r := newController(t, s, wffc, newNode("node-a"), pod, datadir, healthy, healthyPV, occupant) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "an occupant must not be treated as proof of unavailability without any required anti-affinity; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + t.Run("Node lookup uses the uncached Reader, not the cached Client", func(t *testing.T) { + // nodeUnavailableForScheduling must read the candidate PV's + // pinned Node through the uncached Reader (matching + // freedPVBlocking): this controller's RBAC grants Node get and + // list — not watch — so a cache-backed Get here would force + // (and fail without) a cluster-wide Node watch on a real, + // RBAC-restricted install. The cached Client + // below has NO Node object at all, standing in for that + // permission gap; a healthy, existing, uncordoned "node-a" is + // only reachable through Reader. If the code ever regresses to + // reading Nodes off Client, it would see NotFound and wrongly + // treat "node-a" as unavailable, exempting a claim that (per + // the real Node state, visible only via Reader) isn't actually + // mis-pinned. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + r := newController(t, s, wffc, pod, datadir, healthy, healthyPV) // no Node in the cached Client + // The Reader (standing in for fresh API-server state) carries + // the stuck Pod, the PVC evidence, and the healthy Node; the + // Node is the only object whose visibility differs between the + // two clients. + r.Reader = fake.NewClientBuilder().WithScheme(s).WithObjects(pod, newNode("node-a"), datadir, healthy).Build() + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "the healthy Node seen via Reader must not be exempted; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + t.Run("occupant Pod lookup uses the uncached Reader, not a stale cached Client", func(t *testing.T) { + // The occupancy check must read sibling Pods through the + // uncached Reader, exactly like the Node lookup above: if + // occupant rp-0 was ALREADY deleted or rescheduled elsewhere by + // the time this Reconcile runs, but the informer cache hasn't + // caught up yet, trusting the stale cached copy would + // manufacture an anti-affinity conflict that no longer exists — + // on a Node that may by now be perfectly schedulable — and + // destructively delete rp-1's healthy bound claim for nothing. + // Here the cached Client still has rp-0 sitting on "node-a" + // (stale, no DeletionTimestamp, old Spec.NodeName); the + // uncached Reader — standing in for current API-server state — + // has no such Pod at all. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + pod.Spec.Affinity = hardAntiAffinity(map[string]string{operatorlabels.InstanceKey: "redpanda"}) + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + staleOccupant := newPod("rp-0", "ns", "redpanda") + staleOccupant.Spec.NodeName = "node-a" + // The cached Client still has the stale occupant on "node-a"... + r := newController(t, s, wffc, newNode("node-a"), pod, datadir, healthy, healthyPV, staleOccupant) + // ...but the uncached Reader (fresher API-server state) does + // not — it carries the stuck Pod, the PVC evidence, and the + // healthy Node, just no occupant. + r.Reader = fake.NewClientBuilder().WithScheme(s).WithObjects(pod, newNode("node-a"), datadir, healthy).Build() + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a stale cached occupant must not be treated as proof of unavailability; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + t.Run("sibling discovery uses the uncached Reader, not a stale cached Client", func(t *testing.T) { + // The sibling scan feeds ShouldRemediate's Timeout freshness + // check, which is only as fresh as the read behind it: sibling + // rp-0 was genuinely stuck, then got resolved out-of-band (its + // recreated Pod scheduled onto node-b and is Running), leaving + // its datadir claim genuinely mid-settling — exactly what Gate 3 + // must now defer on. The cached Client still serves the OLD + // stuck rp-0 (Pending, volume-affinity failure, mis-pinned bound + // claim — evidence that would fully re-manufacture the + // exemption); the uncached Reader — standing in for current + // API-server state — serves the fresh Running rp-0, which fails + // pvcUnbinderPredicate and earns no exemption. Gate 3 must + // defer on rp-0's unbound claim instead of letting rp-1's unbind + // proceed concurrently with rp-0's in-flight recovery. + pod := withPVC(podWithVolumeAffinityFailure("rp-1", "ns", "redpanda"), "datadir-rp-1") + mispinned := newPVC("datadir-rp-1", "ns", "redpanda", "pv-data-1") + pv := boundHostPathPV("pv-data-1", "datadir-rp-1", "node-a") + + staleSibling := withPVC(withPVC(podWithVolumeAffinityFailure("rp-0", "ns", "redpanda"), "datadir-rp-0"), "shadow-index-cache-rp-0") + freshSibling := newPod("rp-0", "ns", "redpanda") + freshSibling.Status.Phase = corev1.PodRunning + freshSibling.Spec.NodeName = "node-b" + + siblingDatadir := newPVC("datadir-rp-0", "ns", "redpanda", "") + siblingDatadir.Spec.StorageClassName = ptr.To("standard") + siblingShadow := newPVC("shadow-index-cache-rp-0", "ns", "redpanda", "pv-shadow-rp-0") + siblingShadowPV := boundHostPathPV("pv-shadow-rp-0", "shadow-index-cache-rp-0", "node-a") + + cordoned := newNode("node-a") + cordoned.Spec.Unschedulable = true + + // The cached Client still holds the stale stuck rp-0 with its + // full (stale-consistent) exemption evidence... + r := newController(t, s, wffc, pod, mispinned, pv, staleSibling, siblingDatadir, siblingShadow, siblingShadowPV) + // ...while the Reader holds the fresh Running rp-0 plus the + // live PVC/Node evidence. + r.Reader = fake.NewClientBuilder().WithScheme(s).WithObjects( + pod, freshSibling, cordoned, mispinned, siblingDatadir, siblingShadow, + ).Build() + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a stale cached sibling must not re-manufacture the exemption; Gate 3 must defer on the fresh state") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "datadir-rp-1"}, &gotPVC), "rp-1's bound claim must not be deleted") + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("generic scheduling failure with a NotReady/unreachable pinned node is still exempted", func(t *testing.T) { + // A real, common node-loss shape distinct from cordoning: the + // Node object still exists but its kubelet crashed or it's + // network-partitioned, so Ready is False/Unknown. Without + // recognizing this, a stuck pod with an unbound WFFC sibling + // claim would defer at Gate 3 forever instead of unbinding the + // pinned claim — exactly the scenario ShouldRemediate already + // accepts via the "untolerated taint + // {node.kubernetes.io/unreachable: }" scheduler message. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 node(s) had untolerated taint {node.kubernetes.io/unreachable: }.", + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + mispinned := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-mispinned-rp-1") + mispinnedPV := boundHostPathPV("pv-mispinned-rp-1", "shadow-index-cache-rp-1", "node-a") + unreachable := newNode("node-a") + unreachable.Status.Conditions = []corev1.NodeCondition{{Type: corev1.NodeReady, Status: corev1.ConditionUnknown}} + r := newController(t, s, wffc, unreachable, pod, datadir, mispinned, mispinnedPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "a claim pinned to a NotReady/unreachable node must still be exempted") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the claim pinned to the unreachable node must be deleted") + }) + + t.Run("generic scheduling failure with the standard unreachable taint (no Ready condition set) is still exempted", func(t *testing.T) { + // Covers the taint check independently of the Ready-condition + // check (they're normally applied together by the node + // lifecycle controller, but this guards against the lag window + // between the two, and against relying on Ready alone). + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 node(s) had untolerated taint {node.kubernetes.io/unreachable: }.", + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + mispinned := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-mispinned-rp-1") + mispinnedPV := boundHostPathPV("pv-mispinned-rp-1", "shadow-index-cache-rp-1", "node-a") + tainted := newNode("node-a") + tainted.Spec.Taints = []corev1.Taint{{Key: taintNodeUnreachable, Effect: corev1.TaintEffectNoExecute}} + r := newController(t, s, wffc, tainted, pod, datadir, mispinned, mispinnedPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "a claim pinned to a node carrying the unreachable taint must still be exempted") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err)) + }) + + t.Run("pod's default grace-period toleration for the unreachable taint does not make the node look available", func(t *testing.T) { + // Every Pod gets an auto-injected toleration for + // node.kubernetes.io/unreachable with a finite + // TolerationSeconds (the DefaultTolerationSeconds admission + // plugin's eviction grace period) unless it declares its own. + // That grace period is about eviction timing, not about the + // node being fine to (re)schedule onto, and must not cause the + // unreachable node to be treated as tolerated/available. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 node(s) had untolerated taint {node.kubernetes.io/unreachable: }.", + }} + pod.Spec.Tolerations = []corev1.Toleration{{ + Key: taintNodeUnreachable, + Operator: corev1.TolerationOpExists, + Effect: corev1.TaintEffectNoExecute, + TolerationSeconds: ptr.To(int64(300)), + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") + mispinned := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-mispinned-rp-1") + mispinnedPV := boundHostPathPV("pv-mispinned-rp-1", "shadow-index-cache-rp-1", "node-a") + tainted := newNode("node-a") + tainted.Spec.Taints = []corev1.Taint{{Key: taintNodeUnreachable, Effect: corev1.TaintEffectNoExecute}} + r := newController(t, s, wffc, tainted, pod, datadir, mispinned, mispinnedPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "the default grace-period toleration must not make the unreachable node look available") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err)) + }) + + t.Run("pod's unconditional toleration for the unreachable taint leaves it unproven, so Gate 3 keeps deferring", func(t *testing.T) { + // The flip side: with an unconditional (no TolerationSeconds) + // toleration for the taint, and no Ready condition set, the + // taint alone can no longer prove the node is unavailable to + // this Pod — and nothing else does either (node exists, + // uncordoned, unoccupied), so the exemption must not fire. + pod := withPVC(withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + pod.Spec.Tolerations = []corev1.Toleration{{ + Key: taintNodeUnreachable, + Operator: corev1.TolerationOpExists, + Effect: corev1.TaintEffectNoExecute, + }} + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + // WFFC class set so the ONLY thing standing between deferral + // and exemption is the evaluator under test — without it, + // claimUsesWaitForFirstConsumer fails the claim unconditionally + // and the test would pass even with the evaluator broken. + datadir.Spec.StorageClassName = ptr.To("standard") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + tainted := newNode("node-a") + // Both effect twins, as on a real unreachable node — the + // NoSchedule twin must be judged through the same NoExecute + // toleration lens, not its raw effect. + tainted.Spec.Taints = []corev1.Taint{ + {Key: taintNodeUnreachable, Effect: corev1.TaintEffectNoExecute}, + {Key: taintNodeUnreachable, Effect: corev1.TaintEffectNoSchedule}, + } + r := newController(t, s, wffc, tainted, pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "an unconditionally-tolerated taint must not by itself prove the node unavailable") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + t.Run("tolerate-forever mode keeps a transiently NotReady node unproven despite the Ready condition", func(t *testing.T) { + // Under --broker-pod-node-unavailable-toleration=-1s the + // operator injects unconditional (no TolerationSeconds) + // not-ready/unreachable NoExecute tolerations onto broker + // pods, and that mode's documented contract is that only + // Node-object DELETION signals permanent node loss. For such + // pods the exemption must be withheld as a POLICY choice (the + // scheduler might refuse this node right now because of the + // NoSchedule twin taint, but the flag says transient + // unreachability is never permanent loss), so Gate 3 must keep + // deferring instead of escalating the transient partition into + // deleting the pod's bound claim. + pod, datadir, shadow, pv := stuckBroker("rp-1") + pod.Spec.Tolerations = []corev1.Toleration{ + {Key: taintNodeNotReady, Operator: corev1.TolerationOpExists, Effect: corev1.TaintEffectNoExecute}, + {Key: taintNodeUnreachable, Operator: corev1.TolerationOpExists, Effect: corev1.TaintEffectNoExecute}, + } + unreachable := newNode("node-a") + unreachable.Status.Conditions = []corev1.NodeCondition{{Type: corev1.NodeReady, Status: corev1.ConditionUnknown}} + // A real unreachable node carries BOTH effect twins: the node + // lifecycle controller's eviction pass applies NoExecute and + // its condition pass applies NoSchedule, keyed off the same + // Ready condition. The -1s-injected tolerations are + // NoExecute-shaped only, so the NoSchedule twin is exactly the + // shape that must NOT defeat the carve-out. + unreachable.Spec.Taints = []corev1.Taint{ + {Key: taintNodeUnreachable, Effect: corev1.TaintEffectNoExecute}, + {Key: taintNodeUnreachable, Effect: corev1.TaintEffectNoSchedule}, + } + r := newController(t, s, wffc, unreachable, pod, datadir, shadow, pv) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a tolerate-forever pod's NotReady node must not be treated as unavailable; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the bound claim must survive a transiently unreachable node in tolerate-forever mode") + }) + + t.Run("grace-period tolerations do not suppress the Ready-condition proof", func(t *testing.T) { + // The counterpart: the DEFAULT auto-injected tolerations carry + // a finite TolerationSeconds (an eviction grace window, not a + // statement that the node is fine), so with those the + // Ready-condition leg must still prove unavailability and the + // exemption must still fire. + pod, datadir, shadow, pv := stuckBroker("rp-1") + pod.Spec.Tolerations = []corev1.Toleration{ + {Key: taintNodeNotReady, Operator: corev1.TolerationOpExists, Effect: corev1.TaintEffectNoExecute, TolerationSeconds: ptr.To(int64(300))}, + {Key: taintNodeUnreachable, Operator: corev1.TolerationOpExists, Effect: corev1.TaintEffectNoExecute, TolerationSeconds: ptr.To(int64(300))}, + } + unreachable := newNode("node-a") + unreachable.Status.Conditions = []corev1.NodeCondition{{Type: corev1.NodeReady, Status: corev1.ConditionUnknown}} + r := newController(t, s, wffc, unreachable, pod, datadir, shadow, pv) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "grace-period tolerations must not suppress the Ready-condition unavailability proof") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the mis-pinned bound claim must be deleted") + }) + + t.Run("evidence-read failure downgrades to the conservative deferral instead of failing the reconcile", func(t *testing.T) { + // RBAC version skew (operator image upgraded ahead of its + // ClusterRole) surfaces as a 403 on the exemption chain's + // nodes LIST. That must not error-loop the reconcile — which + // would fire neither the gate metric nor the deferral Event — + // but downgrade to the pre-exemption behavior: Gate 3 defers + // on the unbound claim with the usual 30s requeue. + pod, datadir, shadow, pv := stuckBroker("rp-1") + c := fake.NewClientBuilder().WithScheme(s). + WithObjects(wffc, pod, datadir, shadow, pv). + WithInterceptorFuncs(interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*corev1.NodeList); ok { + return apierrors.NewForbidden(schema.GroupResource{Resource: "nodes"}, "", fmt.Errorf("RBAC version skew")) + } + return cl.List(ctx, list, opts...) + }, + }).Build() + r := &Controller{Client: c} + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err, "an evidence-read failure must not fail the reconcile") + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "Gate 3 must fall back to the conservative deferral") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "nothing may be deleted when the evidence chain is unavailable") + }) + + t.Run("evidence reads are skipped entirely when no claim is unbound", func(t *testing.T) { + // The unbinder's original scenario — node dead, ALL claims + // Bound — must not depend on the exemption evidence chain at + // all: with zero unbound claims Gate 3 has nothing to defer + // on, so the (broken, per the interceptor) nodes LIST must + // never run and remediation must proceed exactly as it did + // before the exemption existed. + pod := withPVC(podWithVolumeAffinityFailure("rp-1", "ns", "redpanda"), "datadir-rp-1") + mispinned := newPVC("datadir-rp-1", "ns", "redpanda", "pv-data-1") + pv := boundHostPathPV("pv-data-1", "datadir-rp-1", "node-a") + c := fake.NewClientBuilder().WithScheme(s). + WithObjects(wffc, pod, mispinned, pv). + WithInterceptorFuncs(interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*corev1.NodeList); ok { + return apierrors.NewForbidden(schema.GroupResource{Resource: "nodes"}, "", fmt.Errorf("RBAC version skew")) + } + return cl.List(ctx, list, opts...) + }, + }).Build() + r := &Controller{Client: c} + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err, "the all-claims-bound path must not touch the evidence chain") + require.Zero(t, res.RequeueAfter) + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "datadir-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the mis-pinned bound claim must be deleted despite the broken nodes LIST") + }) + + t.Run("reconciled Pod is re-qualified on the uncached Reader before evidence or destruction", func(t *testing.T) { + // The initial cached Get is only a pre-filter: if the informer + // still serves an OLD rp-1 (stuck past the Timeout, mis-pinned + // claims and all) while the API server already has rp-1 + // resolved (here: Running on node-b after a recreate), the + // stale copy must not supply the exemption evidence and reach + // the PVC deletes — the claim preconditions guard the claims, + // not the Pod evidence that justified deleting them. The + // cached Client below holds the fully-armed stale rp-1; the + // uncached Reader holds the fresh Running rp-1 plus everything + // the stale path would need to succeed, so a regression to + // cached-only qualification destroys the claim and fails this + // test. + stalePod, datadir, shadow, pv := stuckBroker("rp-1") + freshPod := newPod("rp-1", "ns", "redpanda") + freshPod.Status.Phase = corev1.PodRunning + freshPod.Spec.NodeName = "node-b" + cordoned := newNode("node-a") + cordoned.Spec.Unschedulable = true + r := newController(t, s, wffc, stalePod, datadir, shadow, pv) + r.Reader = fake.NewClientBuilder().WithScheme(s).WithObjects( + freshPod, datadir, shadow, cordoned, wffc, + ).Build() + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(stalePod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "a Pod that no longer qualifies on fresh state is simply skipped") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "no claim may be deleted on stale Pod evidence") + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod), "the Pod must not be deleted on stale evidence") + }) + + t.Run("a sibling's deadlock proof does not authorize destroying a Pod without its own mis-pin proof", func(t *testing.T) { + // Exemptions exist to break deadlocks, not to transitively + // unlock unrelated destruction: rp-0 is genuinely deadlocked + // (its shadow PV pinned to gone node-a, unbound WFFC datadir), + // while the reconciled rp-1 is Pending on plain CPU pressure + // with a healthy bound claim whose multi-value NodeAffinity + // [node-a, node-b] still has available node-b — rp-1's own + // mis-pin proof fails. rp-0's exemption clears the only + // unbound claim, which pre-exemption would have incidentally + // deferred rp-1's destruction; the post-exemption own-proof + // check must restore that deferral instead of deleting rp-1's + // healthy claim. (Both PVs pin node-a as Values[0], so Gate + // 2's distinct-node count stays at one and does not mask the + // property under test.) + sibling, siblingDatadir, siblingShadow, siblingPV := stuckBroker("rp-0") + pod := withPVC(newPod("rp-1", "ns", "redpanda"), "datadir-rp-1") + pod.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 Insufficient cpu.", + }} + healthy := newPVC("datadir-rp-1", "ns", "redpanda", "pv-data-1") + healthyPV := boundHostPathPV("pv-data-1", "datadir-rp-1", "node-a") + healthyPV.Spec.NodeAffinity.Required.NodeSelectorTerms[0].MatchExpressions[0].Values = []string{"node-a", "node-b"} + // node-a is gone entirely (rp-0's proof); node-b is present + // and healthy (defeats rp-1's own proof). + r := newController(t, s, wffc, newNode("node-b"), pod, healthy, healthyPV, sibling, siblingDatadir, siblingShadow, siblingPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "the sibling's exemption must not authorize destroying rp-1 without rp-1's own mis-pin proof") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "datadir-rp-1"}, &gotPVC), "rp-1's healthy bound claim must not be deleted") + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("own unbound claim with explicit empty storageClassName does not exempt a healthy bound claim", func(t *testing.T) { + // storageClassName: "" is a distinct, explicit Kubernetes shape + // ("no storage class, no dynamic provisioning" — what the + // redpanda chart produces via `storage.persistentVolume.storageClass: "-"`). + // With no beta annotation present (the annotation wins whenever + // its key exists; Spec.StorageClassName applies only when it is + // absent, as here), the explicit "" must resolve to "no class" + // — even though a WaitForFirstConsumer StorageClass exists + // elsewhere in the cluster, it must NOT be consulted for this + // claim. "shadow-index-cache" is a healthy Bound HostPath/Local + // claim that must survive. + pod := withPVC(withPVC(podWithVolumeAffinityFailure("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("") + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + r := newController(t, s, wffc, pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "an explicit empty storageClassName must resolve to no class and not be exempted; Gate 3 must keep deferring") + + // Nothing was touched — in particular the healthy bound claim + // must survive; it must not be deleted just because an + // unrelated WaitForFirstConsumer StorageClass happens to exist + // in the cluster. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("nil StorageClassName does not fall back to the cluster's current WFFC-mode StorageClass", func(t *testing.T) { + // This is the finding an adversarial review flagged: a claim + // left with Spec.StorageClassName == nil and no beta annotation + // must resolve to "no class", NOT to whatever StorageClass + // currently happens to be present/marked default in the + // cluster. Kubernetes' own DefaultStorageClass admission + // controller mutates Spec.StorageClassName onto the PERSISTED + // object at CREATE time — the one and only time defaulting + // happens — so a still-nil field on an existing claim means + // admission found no default to apply back then (or was + // disabled), not "consult today's default instead". "wffc" here + // stands in for a StorageClass that exists in the cluster but + // must NOT be treated as governing this claim. + pod := withPVC(withPVC(podWithVolumeAffinityFailure("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") // Spec.StorageClassName left nil, no annotation + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + r := newController(t, s, wffc, pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a nil StorageClassName must not fall back to a StorageClass that merely exists in the cluster; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + t.Run("beta storage-class annotation pointing at a non-WFFC class does not exempt despite a WFFC class existing", func(t *testing.T) { + // A legacy PVC (or one created by a client that still writes + // the pre-1.6 annotation instead of Spec.StorageClassName) is + // governed by that annotation, exactly as the PV controller's + // own GetPersistentVolumeClaimClass resolves it. Here the + // annotation names an Immediate-mode class — a real + // provisioning failure, not the WaitForFirstConsumer deadlock + // this exemption targets — even though "wffc" (unreferenced) + // also exists in the cluster. + immediate := storagev1.VolumeBindingImmediate + immediateSC := &storagev1.StorageClass{ + ObjectMeta: metav1.ObjectMeta{Name: "immediate-sc"}, + VolumeBindingMode: &immediate, + } + pod := withPVC(withPVC(podWithVolumeAffinityFailure("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Annotations = map[string]string{corev1.BetaStorageClassAnnotation: "immediate-sc"} + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + r := newController(t, s, wffc, immediateSC, pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "the beta-annotated Immediate-mode class must govern, not the unrelated WFFC class; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + t.Run("beta storage-class annotation pointing at a WFFC class is honored and still exempted", func(t *testing.T) { + // The positive counterpart: with no Spec.StorageClassName set, + // the beta annotation naming a WaitForFirstConsumer class must + // actually be resolved (not silently ignored) for the + // exemption to be granted where Kubernetes itself would + // consider the claim WaitForFirstConsumer. + pod, datadir, mispinned, mispinnedPV := stuckBroker("rp-1") + datadir.Spec.StorageClassName = nil + datadir.Annotations = map[string]string{corev1.BetaStorageClassAnnotation: "standard"} + r := newController(t, s, wffc, pod, datadir, mispinned, mispinnedPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "a beta-annotated WaitForFirstConsumer class must be honored and exempted") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the mis-pinned bound claim must be deleted") + }) + + t.Run("conflicting spec and annotation: annotation names Immediate, Spec names WFFC — annotation governs, not exempted", func(t *testing.T) { + // Kubernetes' own component-helpers/storage/volume.GetPersistentVolumeClaimClass + // (vendored at this module's pinned v0.35.1) checks the beta + // annotation FIRST — whenever the key is present at all — and + // only consults Spec.StorageClassName when the annotation key + // is entirely absent. That's the reverse of what might seem + // intuitive (the modern Spec field "should" win), but a PVC + // carrying both must resolve via the annotation to match how + // the PV controller actually binds it. Here Spec names a WFFC + // class but the annotation names an Immediate one — the + // annotation must govern, so this must NOT be exempted. + immediate := storagev1.VolumeBindingImmediate + immediateSC := &storagev1.StorageClass{ + ObjectMeta: metav1.ObjectMeta{Name: "immediate-sc"}, + VolumeBindingMode: &immediate, + } + pod := withPVC(withPVC(podWithVolumeAffinityFailure("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") // WFFC + datadir.Annotations = map[string]string{corev1.BetaStorageClassAnnotation: "immediate-sc"} + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + r := newController(t, s, wffc, immediateSC, pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "the annotation's Immediate-mode class must govern over a conflicting WFFC Spec.StorageClassName; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + t.Run("conflicting spec and annotation: annotation names WFFC, Spec names Immediate — annotation governs, still exempted", func(t *testing.T) { + // The mirror image of the case above: the annotation names the + // WFFC class while Spec names an unrelated Immediate-mode + // class. The annotation still governs, so this must be exempted. + immediate := storagev1.VolumeBindingImmediate + immediateSC := &storagev1.StorageClass{ + ObjectMeta: metav1.ObjectMeta{Name: "immediate-sc"}, + VolumeBindingMode: &immediate, + } + pod, datadir, mispinned, mispinnedPV := stuckBroker("rp-1") + datadir.Spec.StorageClassName = ptr.To("immediate-sc") + datadir.Annotations = map[string]string{corev1.BetaStorageClassAnnotation: "standard"} + r := newController(t, s, wffc, immediateSC, pod, datadir, mispinned, mispinnedPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "the annotation's WFFC class must govern over a conflicting Immediate-mode Spec.StorageClassName") + + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err), "the mis-pinned bound claim must be deleted") + }) + + t.Run("present-but-empty annotation overrides a WFFC Spec.StorageClassName", func(t *testing.T) { + // A present annotation with an empty string value still counts + // as "present" for Kubernetes' own precedence — it resolves to + // "no storage class", and still wins over Spec.StorageClassName + // even when the Spec names a real WFFC class. Getting this + // wrong (treating a present-but-empty annotation as "absent, + // fall back to Spec") would wrongly exempt this claim. + pod := withPVC(withPVC(podWithVolumeAffinityFailure("rp-1", "ns", "redpanda"), "datadir-rp-1"), "shadow-index-cache-rp-1") + datadir := newPVC("datadir-rp-1", "ns", "redpanda", "") + datadir.Spec.StorageClassName = ptr.To("standard") // WFFC + datadir.Annotations = map[string]string{corev1.BetaStorageClassAnnotation: ""} + healthy := newPVC("shadow-index-cache-rp-1", "ns", "redpanda", "pv-healthy-rp-1") + healthyPV := boundHostPathPV("pv-healthy-rp-1", "shadow-index-cache-rp-1", "node-a") + r := newController(t, s, wffc, pod, datadir, healthy, healthyPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a present-but-empty annotation must resolve to no class and override Spec.StorageClassName; Gate 3 must keep deferring") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "the healthy bound claim must not be deleted") + }) + + t.Run("allow-pv-rebinding keeps the conservative deferral even for own claims", func(t *testing.T) { + // Under --allow-pv-rebinding the unbinder floats freed PVs as + // live binding candidates (see FreedPVAnnotation); acting while + // ANY claim is unbound risks pairing that claim with a freed + // disk it was never meant to hold (the INC-2818 cross-claim + // swap, intra-pod variant). The exemption must not apply. + pod, datadir, shadow, pv := stuckBroker("rp-1") + r := newController(t, s, wffc, pod, datadir, shadow, pv) + r.AllowRebinding = true + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "rebinding mode must keep deferring on any unbound claim") + + // Nothing was touched. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC)) + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("DisableStuckClaimExemption kill switch restores the conservative deferral", func(t *testing.T) { + // The escape hatch for environments where the proof chain + // misfires: with the flag set, the exemption never runs and + // Gate 3 defers on every unbound claim (the pre-exemption + // behavior), while the rest of the unbinder keeps working. + pod, datadir, shadow, pv := stuckBroker("rp-1") + r := newController(t, s, wffc, pod, datadir, shadow, pv) + r.DisableStuckClaimExemption = true + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "the kill switch must restore deferral on every unbound claim") + + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC), "nothing may be deleted with the exemption disabled") + }) + + t.Run("mis-pin proof requires the PV to reference the claim back", func(t *testing.T) { + // spec.volumeName is user-settable at claim creation (static + // pre-binding), so a claim pointing at a PV is not evidence of + // a binding. Only the binder completes the two-way link by + // stamping the PV's ClaimRef with the claim's UID. A PV whose + // ClaimRef is missing or references a different claim must not + // serve as mis-pin proof — otherwise a forged claim pre-pointed + // at an arbitrary local PV could waive Gate 3. + for name, mutate := range map[string]func(*corev1.PersistentVolume){ + "nil ClaimRef": func(pv *corev1.PersistentVolume) { pv.Spec.ClaimRef = nil }, + "wrong name": func(pv *corev1.PersistentVolume) { pv.Spec.ClaimRef.Name = "some-other-claim" }, + "mismatched UID": func(pv *corev1.PersistentVolume) { pv.Spec.ClaimRef.UID = "uid-of-someone-else" }, + } { + t.Run(name, func(t *testing.T) { + pod, datadir, shadow, pv := stuckBroker("rp-1") + mutate(pv) + r := newController(t, s, wffc, pod, datadir, shadow, pv) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a PV without a matching ClaimRef back-reference must not prove a mis-pin") + + // Nothing was touched. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC)) + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + } + }) + + t.Run("exemption paper trail is not recorded while the freed-pv gate still defers", func(t *testing.T) { + // The freed-pv gate (Gate 4) runs AFTER the exemption and its + // durable annotation can hold for days. The exempted + // metric/Event claim "proceeded past the pvc-rebinding gate" — + // recording them on a reconcile that then defers at freed-pv + // would count a gate pass every 30s while nothing proceeds. + pod, datadir, shadow, pv := stuckBroker("rp-1") + freed := pvWithAnnotations("pv-freed", corev1.VolumeAvailable, "node-live", map[string]string{ + FreedPVAnnotation: "/ns/redpanda", + }) + recorder := &events.FakeRecorder{Events: make(chan string, 8)} + r := newController(t, s, wffc, pod, datadir, shadow, pv, freed, newNode("node-live")) + r.Recorder = recorder + + exemptedBefore := promtestutil.ToFloat64(observability.PVCUnbinderGateExempted) + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "the freed-pv gate must still defer") + require.Equal(t, exemptedBefore, promtestutil.ToFloat64(observability.PVCUnbinderGateExempted), "the exempted metric must not count a gate pass that never proceeded") + + // The paper trail names the freed-pv deferral, never a gate + // pass that didn't happen. + close(recorder.Events) + for ev := range recorder.Events { + require.NotContains(t, ev, eventReasonGateExempted, "no exemption event may be recorded on a reconcile that defers at the freed-pv gate") + } + + // Nothing was touched. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC)) + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("sibling stuck on a generic scheduling failure without a mis-pinned bound claim still defers", func(t *testing.T) { + // The weak signature (schedulingFailureRE) alone is not enough + // to grant the exemption: a sibling Pending on "0/N nodes are + // available: ... unbound ... PersistentVolumeClaims" could + // equally be stuck on an unrelated storage-class, provisioner, + // or quota failure rather than the provably-deadlocked + // mis-pinned-bound-claim pattern. Without a Bound claim on a + // HostPath/Local PV with NodeAffinity to prove the deadlock, + // Gate 3 must keep deferring. + pod1, datadir1, shadow1, pv1 := stuckBroker("rp-1") + sibling := withPVC(newPod("rp-0", "ns", "redpanda"), "datadir-rp-0") + sibling.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.", + }} + siblingClaim := newPVC("datadir-rp-0", "ns", "redpanda", "") + r := newController(t, s, wffc, pod1, datadir1, shadow1, pv1, sibling, siblingClaim) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod1)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a sibling with no proven mis-pinned bound claim must still defer the unbind") + + // Nothing was touched. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC)) + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("sibling stuck on a non-affinity total scheduling failure with a mis-pinned bound claim is still exempted", func(t *testing.T) { + // Pins the deliberate breadth of the *signature* match: modern + // schedulers prefix every total scheduling failure with "0/N + // nodes are available:" and don't reliably name volume + // affinity. A sibling Pending on that weaker message form is + // still exempted, as long as it also has independent proof of + // the deadlock shape — a Bound claim on a HostPath/Local PV + // with NodeAffinity. + pod1, datadir1, shadow1, pv1 := stuckBroker("rp-1") + sibling := withPVC(withPVC(newPod("rp-0", "ns", "redpanda"), "datadir-rp-0"), "shadow-index-cache-rp-0") + sibling.Status.Conditions = []corev1.PodCondition{{ + Type: corev1.PodScheduled, + Status: corev1.ConditionFalse, + Reason: "Unschedulable", + Message: "0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.", + }} + siblingClaim := newPVC("datadir-rp-0", "ns", "redpanda", "") + siblingClaim.Spec.StorageClassName = ptr.To("standard") + siblingShadow := newPVC("shadow-index-cache-rp-0", "ns", "redpanda", "pv-shadow-rp-0") + siblingPV := boundHostPathPV("pv-shadow-rp-0", "shadow-index-cache-rp-0", "node-a") + r := newController(t, s, wffc, pod1, datadir1, shadow1, pv1, sibling, siblingClaim, siblingShadow, siblingPV) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod1)}) + require.NoError(t, err) + require.Zero(t, res.RequeueAfter, "a sibling matching the weak stuck signature with a proven mis-pinned bound claim must not defer the unbind") + + // Only rp-1's objects were touched. + var gotPVC corev1.PersistentVolumeClaim + err = r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC) + require.True(t, apierrors.IsNotFound(err)) + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "datadir-rp-0"}, &gotPVC)) + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-0"}, &gotPVC)) + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-0"}, &gotPod)) + }) + + t.Run("sibling whose Unschedulable condition is fresher than r.Timeout still defers", func(t *testing.T) { + // ShouldRemediate requires a Pod's Unschedulable condition to + // be at least r.Timeout old before treating it as confirmed + // stuck rather than a transient scheduler/provisioner hiccup — + // the reconciled Pod itself is gated on this before Reconcile + // ever reaches Gate 3. A sibling must earn that same + // confirmation, not just match the weak signature and have a + // mis-pinned bound claim: rp-0 here turned Unschedulable only + // moments ago (LastTransitionTime ~= now, well under + // r.Timeout), so its unbound claim must NOT be exempted yet — + // Gate 0 provides no backstop here, since no prior unbind has + // annotated a PV for a sibling that's merely freshly stuck. + pod1, datadir1, shadow1, pv1 := stuckBroker("rp-1") + sibling, siblingDatadir, siblingShadow, siblingPV := stuckBroker("rp-0") + sibling.Status.Conditions[0].LastTransitionTime = metav1.Now() + r := newController(t, s, wffc, pod1, datadir1, shadow1, pv1, sibling, siblingDatadir, siblingShadow, siblingPV) + r.Timeout = 30 * time.Second + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod1)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a sibling fresher than r.Timeout must not be exempted; Gate 3 must keep deferring") + + // Nothing was touched. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC)) + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("sibling excluded by Selector still defers despite a mis-pinned bound claim", func(t *testing.T) { + // The Selector is an operational boundary: an admin can scope + // this controller to a subset of Pods. A stuck sibling outside + // that scope must not be treated as exempt-worthy, or the + // Selector stops meaningfully limiting which claims Gate 3 + // treats as safe to bypass. + pod1, datadir1, shadow1, pv1 := stuckBroker("rp-1") + pod1.Labels["scope"] = "in" + sibling, siblingDatadir, siblingShadow, siblingPV := stuckBroker("rp-0") + sibling.Labels["scope"] = "out" + r := newController(t, s, wffc, pod1, datadir1, shadow1, pv1, sibling, siblingDatadir, siblingShadow, siblingPV) + r.Selector = labels.SelectorFromSet(labels.Set{"scope": "in"}) + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod1)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a Selector-excluded sibling must still defer the unbind") + + // Nothing was touched. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "shadow-index-cache-rp-1"}, &gotPVC)) + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) + + t.Run("sibling unbound claim still defers", func(t *testing.T) { + // The serialization Gate 3 exists for: another broker's claim + // is mid-rebind (unbound, owner pod absent or not stuck on + // volume affinity), so rp-1's unbind must wait. Only claims of + // provably-stuck pods are exempt. + pod := withPVC(podWithVolumeAffinityFailure("rp-1", "ns", "redpanda"), "datadir-rp-1") + mispinned := newPVC("datadir-rp-1", "ns", "redpanda", "pv-data-1") + pv := boundHostPathPV("pv-data-1", "datadir-rp-1", "node-a") + sibling := newPVC("datadir-rp-0", "ns", "redpanda", "") + recorder := &events.FakeRecorder{Events: make(chan string, 8)} + r := newController(t, s, wffc, pod, mispinned, sibling, pv) + r.Recorder = recorder + + res, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(pod)}) + require.NoError(t, err) + require.Equal(t, requeueDuringDisruption, res.RequeueAfter, "a sibling's unbound claim must still defer the unbind") + + // The pvc-rebinding gate specifically fired (not another gate + // returning the same requeue), and the event names the claim + // that gated the unbind. + select { + case ev := <-recorder.Events: + require.Contains(t, ev, "gate="+gatePVCRebinding) + require.Contains(t, ev, "datadir-rp-0") + default: + t.Fatal("expected a PVCUnbinderDeferred event naming the pvc-rebinding gate") + } + + // Nothing was touched. + var gotPVC corev1.PersistentVolumeClaim + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "datadir-rp-1"}, &gotPVC)) + var gotPod corev1.Pod + require.NoError(t, r.Client.Get(ctx, client.ObjectKey{Namespace: "ns", Name: "rp-1"}, &gotPod)) + }) +} + // TestPrepareForUnbind verifies the pre-deletion patch that makes the // in-flight gate durable: Retain policy + both in-flight annotations // (cluster key and claim namespace/name/uid) must land in one call, @@ -685,10 +2410,12 @@ func TestIsClusterPaused(t *testing.T) { } // withPVC adds a StatefulSet-style PVC volume to a Pod (claim name ends -// in pod name, matching StsPVCs() suffix-detection). +// in pod name, matching StsPVCs() suffix-detection). The volume is +// named after the claim so that pods built with multiple calls remain +// valid API objects (volume names must be unique within a pod). func withPVC(p *corev1.Pod, claimName string) *corev1.Pod { p.Spec.Volumes = append(p.Spec.Volumes, corev1.Volume{ - Name: "data", + Name: claimName, VolumeSource: corev1.VolumeSource{ PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{ ClaimName: claimName, @@ -974,6 +2701,206 @@ func TestNodeFromPVAffinity(t *testing.T) { } } +// nodeSelectorTerm is a small helper for building +// corev1.NodeSelectorTerm literals below without repeating the nested +// struct shape. +func nodeSelectorTerm(exprs ...corev1.NodeSelectorRequirement) corev1.NodeSelectorTerm { + return corev1.NodeSelectorTerm{MatchExpressions: exprs} +} + +// hostnameIn builds the standard "kubernetes.io/hostname In [values]" +// NodeSelectorRequirement Local/HostPath PVs use. +func hostnameIn(values ...string) corev1.NodeSelectorRequirement { + return corev1.NodeSelectorRequirement{ + Key: corev1.LabelHostname, + Operator: corev1.NodeSelectorOpIn, + Values: values, + } +} + +// TestPVPinnedHostnames verifies the strict (fail-closed) NodeAffinity +// resolver backing podHasMispinnedBoundClaim's mis-pin proof — unlike +// nodeFromPVAffinity (which nearby TestNodeFromPVAffinity covers), +// this one must resolve the FULL eligible-Node set or explicitly +// refuse, since collapsing to a partial set here would risk unbinding +// a claim that could still bind on a Node it silently dropped. +func TestPVPinnedHostnames(t *testing.T) { + cases := []struct { + name string + pv *corev1.PersistentVolume + want []string + wantOK bool + }{ + { + name: "single term, single value resolves", + pv: newPVWithAffinity("pv", "ns", "claim", "node-a"), + want: []string{"node-a"}, + wantOK: true, + }, + { + name: "single term, multiple values all resolve", + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{ + NodeAffinity: &corev1.VolumeNodeAffinity{ + Required: &corev1.NodeSelector{ + NodeSelectorTerms: []corev1.NodeSelectorTerm{ + nodeSelectorTerm(hostnameIn("node-a", "node-b")), + }, + }, + }, + }}, + want: []string{"node-a", "node-b"}, + wantOK: true, + }, + { + name: "multiple OR'd terms union their hostnames", + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{ + NodeAffinity: &corev1.VolumeNodeAffinity{ + Required: &corev1.NodeSelector{ + NodeSelectorTerms: []corev1.NodeSelectorTerm{ + nodeSelectorTerm(hostnameIn("node-a")), + nodeSelectorTerm(hostnameIn("node-b")), + }, + }, + }, + }}, + want: []string{"node-a", "node-b"}, + wantOK: true, + }, + { + name: "no NodeAffinity fails closed", + pv: &corev1.PersistentVolume{}, + wantOK: false, + }, + { + name: "NodeAffinity without Required fails closed", + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{NodeAffinity: &corev1.VolumeNodeAffinity{}}}, + wantOK: false, + }, + { + name: "empty NodeSelectorTerms fails closed", + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{ + NodeAffinity: &corev1.VolumeNodeAffinity{Required: &corev1.NodeSelector{}}, + }}, + wantOK: false, + }, + { + name: "a term with an additional co-ANDed expression fails closed (can't know the real eligible set)", + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{ + NodeAffinity: &corev1.VolumeNodeAffinity{ + Required: &corev1.NodeSelector{ + NodeSelectorTerms: []corev1.NodeSelectorTerm{ + nodeSelectorTerm( + hostnameIn("node-a"), + corev1.NodeSelectorRequirement{ + Key: "topology.kubernetes.io/zone", + Operator: corev1.NodeSelectorOpIn, + Values: []string{"us-east-1a"}, + }, + ), + }, + }, + }, + }}, + wantOK: false, + }, + { + name: "a term with MatchFields fails closed", + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{ + NodeAffinity: &corev1.VolumeNodeAffinity{ + Required: &corev1.NodeSelector{ + NodeSelectorTerms: []corev1.NodeSelectorTerm{{ + MatchExpressions: []corev1.NodeSelectorRequirement{hostnameIn("node-a")}, + MatchFields: []corev1.NodeSelectorRequirement{{ + Key: "metadata.name", + Operator: corev1.NodeSelectorOpIn, + Values: []string{"node-a"}, + }}, + }}, + }, + }, + }}, + wantOK: false, + }, + { + name: "non-hostname key fails closed (e.g. zone topology)", + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{ + NodeAffinity: &corev1.VolumeNodeAffinity{ + Required: &corev1.NodeSelector{ + NodeSelectorTerms: []corev1.NodeSelectorTerm{ + nodeSelectorTerm(corev1.NodeSelectorRequirement{ + Key: "topology.kubernetes.io/zone", + Operator: corev1.NodeSelectorOpIn, + Values: []string{"us-east-1a"}, + }), + }, + }, + }, + }}, + wantOK: false, + }, + { + name: "hostname NotIn operator fails closed (only In is a positive enumeration)", + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{ + NodeAffinity: &corev1.VolumeNodeAffinity{ + Required: &corev1.NodeSelector{ + NodeSelectorTerms: []corev1.NodeSelectorTerm{ + nodeSelectorTerm(corev1.NodeSelectorRequirement{ + Key: corev1.LabelHostname, + Operator: corev1.NodeSelectorOpNotIn, + Values: []string{"node-a"}, + }), + }, + }, + }, + }}, + wantOK: false, + }, + { + name: "hostname In with no values fails closed", + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{ + NodeAffinity: &corev1.VolumeNodeAffinity{ + Required: &corev1.NodeSelector{ + NodeSelectorTerms: []corev1.NodeSelectorTerm{ + nodeSelectorTerm(hostnameIn()), + }, + }, + }, + }}, + wantOK: false, + }, + { + name: "one resolvable term plus one unresolvable term fails closed for the WHOLE PV", + // Terms are OR'd: if the unresolvable term's real eligible + // set is unknown, the PV's true eligible-Node set can't be + // bounded even though the OTHER term resolved fine. + pv: &corev1.PersistentVolume{Spec: corev1.PersistentVolumeSpec{ + NodeAffinity: &corev1.VolumeNodeAffinity{ + Required: &corev1.NodeSelector{ + NodeSelectorTerms: []corev1.NodeSelectorTerm{ + nodeSelectorTerm(hostnameIn("node-a")), + nodeSelectorTerm(corev1.NodeSelectorRequirement{ + Key: "topology.kubernetes.io/zone", + Operator: corev1.NodeSelectorOpIn, + Values: []string{"us-east-1a"}, + }), + }, + }, + }, + }}, + wantOK: false, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, ok := pvPinnedHostnames(tc.pv) + require.Equal(t, tc.wantOK, ok) + if tc.wantOK { + require.Equal(t, tc.want, got) + } + }) + } +} + // TestListClusterPVCsByName verifies the PVC snapshot helper used by // Gate 0 (cache-staleness tracker check) and Gate 3 (recreated-but-not- // yet-bound detector). Scoping is by namespace AND @@ -986,7 +2913,7 @@ func TestListClusterPVCsByName(t *testing.T) { t.Run("no PVCs returns empty map", func(t *testing.T) { pod := newPod("rp-0", "ns", "redpanda") r := newController(t, s) - got, err := r.listClusterPVCsByName(ctx, pod) + got, err := r.listClusterPVCsByName(ctx, r.Client, pod) require.NoError(t, err) require.NotNil(t, got) require.Empty(t, got) @@ -998,7 +2925,7 @@ func TestListClusterPVCsByName(t *testing.T) { want1 := newPVC("datadir-rp-1", "ns", "redpanda-a", "pv-1") other := newPVC("datadir-rpb-0", "ns", "redpanda-b", "pv-2") r := newController(t, s, want0, want1, other) - got, err := r.listClusterPVCsByName(ctx, pod) + got, err := r.listClusterPVCsByName(ctx, r.Client, pod) require.NoError(t, err) require.Len(t, got, 2) require.Contains(t, got, "datadir-rp-0") @@ -1011,7 +2938,7 @@ func TestListClusterPVCsByName(t *testing.T) { bound := newPVC("datadir-rp-0", "ns", "redpanda", "pv-0") unbound := newPVC("datadir-rp-1", "ns", "redpanda", "") r := newController(t, s, bound, unbound) - got, err := r.listClusterPVCsByName(ctx, pod) + got, err := r.listClusterPVCsByName(ctx, r.Client, pod) require.NoError(t, err) require.Equal(t, "pv-0", got["datadir-rp-0"].Spec.VolumeName) require.Equal(t, "", got["datadir-rp-1"].Spec.VolumeName) @@ -1021,7 +2948,7 @@ func TestListClusterPVCsByName(t *testing.T) { pod := newPod("rp-0", "ns-a", "redpanda") other := newPVC("datadir-rp-0", "ns-b", "redpanda", "pv-0") r := newController(t, s, other) - got, err := r.listClusterPVCsByName(ctx, pod) + got, err := r.listClusterPVCsByName(ctx, r.Client, pod) require.NoError(t, err) require.Empty(t, got) }) @@ -1030,9 +2957,40 @@ func TestListClusterPVCsByName(t *testing.T) { pod := newPod("orphan-0", "ns", "") other := newPVC("datadir-other-0", "ns", "redpanda", "pv-0") r := newController(t, s, other) - got, err := r.listClusterPVCsByName(ctx, pod) + got, err := r.listClusterPVCsByName(ctx, r.Client, pod) require.NoError(t, err) require.NotNil(t, got) require.Empty(t, got) }) } + +// TestClaimListForEvent pins the Event-note cap: events.k8s.io/v1 +// rejects notes over 1024 characters and the broadcaster silently +// drops the rejected Event, so large exempted-claim lists must be +// truncated for Events (logs keep the full list). +func TestClaimListForEvent(t *testing.T) { + short := []string{"datadir-rp-0", "datadir-rp-1"} + require.Equal(t, "[datadir-rp-0 datadir-rp-1]", claimListForEvent(short)) + + var long []string + for i := range 40 { + long = append(long, fmt.Sprintf("shadow-index-cache-redpanda-cluster-%d", i)) + } + capped := claimListForEvent(long) + require.Contains(t, capped, "(+32 more)") + require.Less(t, len(capped), 1024, "a capped note must fit the events.k8s.io/v1 1024-character limit") + msg := fmt.Sprintf("unbound claims %s are exempted as stuck-Pod claims and the reconciled Pod holds its own mis-pin proof; proceeding past the pvc-rebinding gate", capped) + require.Less(t, len(msg), 1024, "the full Event note must fit the limit even with the surrounding message") + + // A count cap alone would not bound the note: claim names can + // legally reach 253 characters, so the cap must also budget total + // rendered length. + var maximal []string + for i := range 10 { + maximal = append(maximal, fmt.Sprintf("%0253d", i)) + } + capped = claimListForEvent(maximal) + msg = fmt.Sprintf("unbound claims %s are exempted as stuck-Pod claims and the reconciled Pod holds its own mis-pin proof; proceeding past the pvc-rebinding gate", capped) + require.Less(t, len(msg), 1024, "maximal-length claim names must still fit the Event note limit") + require.Contains(t, capped, "more)", "the omitted-name count must still be reported") +} diff --git a/operator/internal/observability/metrics.go b/operator/internal/observability/metrics.go index ff91b63e6..e0064506c 100644 --- a/operator/internal/observability/metrics.go +++ b/operator/internal/observability/metrics.go @@ -78,11 +78,24 @@ var ( // detect silent inaction — e.g., a forgotten pause annotation, a // stuck multi-node-event signal, or a cache-staleness hold that // never clears. The `gate` label values are: "pause" (Gate 1), - // "multi-node" (Gate 2), "in-flight" (Gate 0 cache-staleness - // bridge), "pvc-rebinding" (Gate 3, a PVC in the cluster is - // recreated but not yet bound), and "freed-pv" (Gate 4, a PV whose + // "multi-node" (Gate 2), "in-flight" (Gate 0, a previous + // unbind for the cluster has not settled yet), "pvc-rebinding" (Gate 3, a PVC in the cluster is + // recreated but not yet bound — except claims whose Pods are + // PROVABLY deadlocked on a mis-pinned local-PV claim per the + // unbinder's stuckClaimNames proof chain; other stuck Pods, e.g. + // under soft anti-affinity or required terms the unbinder cannot + // interpret, non-WaitForFirstConsumer + // classes, or unresolvable PV node affinity, still increment it, + // so do NOT exclude stuck Pods from alerts on this gate; under + // --allow-pv-rebinding or --disable-pvc-rebinding-gate-exemption + // the exemption is disabled entirely and every unbound claim + // counts), and "freed-pv" (Gate 4, a PV whose // ClaimRef we cleared under --allow-pv-rebinding is still Available // with a live node — unbinding more pods could mis-pair disks). + // Note: the "multi-node" gate also holds the known unfixed sibling + // of the mis-pinned-claim deadlock — when two victims' PVs land on + // two different occupied nodes, the unbinder defers there and + // manual PVC deletion is required. PVCUnbinderGateDeferred = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: metricsNamespace, Subsystem: metricsSubsystem, @@ -90,6 +103,19 @@ var ( Help: "PVCUnbinder reconciles that returned early because a safety gate deferred remediation, labeled by which gate fired.", }, []string{"gate"}) + // PVCUnbinderGateExempted counts reconciles where the pvc-rebinding + // gate (Gate 3) was PASSED because every unbound claim was exempted + // as a stuck-Pod claim — i.e. the unbinder overrode a safety gate + // and proceeded to destructive remediation. This is the metric + // counterpart of the PVCUnbinderGateExempted Event; alert or trend + // on it to notice a mis-firing exemption even after Events age out. + PVCUnbinderGateExempted = prometheus.NewCounter(prometheus.CounterOpts{ + Namespace: metricsNamespace, + Subsystem: metricsSubsystem, + Name: "pvc_unbinder_gate_exempted_total", + Help: "PVCUnbinder reconciles that proceeded past the pvc-rebinding gate because all unbound claims were exempted as stuck-Pod claims.", + }) + // MaintenanceModeCleared counts brokers whose stuck maintenance-mode flag // the operator cleared because the broker had been down (pod not-Ready) // past the configured threshold. A broker left in maintenance mode is @@ -220,6 +246,7 @@ func init() { ReconcileSteadyStateTotal, ReconcileLastSuccessTimestampSeconds, PVCUnbinderGateDeferred, + PVCUnbinderGateExempted, MaintenanceModeCleared, MaintenanceModeClearSkippedAmbiguous,