Skip to content

bug: CreateVolume doesn't set PersistentVolume.spec.nodeAffinity, so a bound PVC's pod can land on any node #403

Closed

Description

@boddumanohar

Summary

The CSI driver's CreateVolume never populates PersistentVolume.spec.nodeAffinity. Once a PVC is bound, a pod referencing it can be scheduled onto any node in the cluster — not just the node the AccessibleTopology/AllowedTopologies gate originally selected at first-consumer binding time.

Why this was harmless until now

For a plain NVMe-oF-backed volume, this genuinely didn't matter: the client just connects over the fabric from wherever the pod lands, and behaves identically from any node. AllowedTopologies + WaitForFirstConsumer only ever needed to influence the initial provisioning decision.

Why it matters now

Found while validating client-side compression/dedup via VDO (issue #277, PR #402). VDO's state (the LVM PV/VG/LV stack) is genuinely node-local — it only exists on whichever node created it. If a pod using a VDO-backed PVC gets rescheduled to a different node (a plain pod delete+recreate is enough; no drain or node failure needed), the CSI driver on that new node has no VDO stack to attach to.

Reproduction

  1. Create a Pool with clientCompression/clientDeduplication enabled (or any StorageClass using allowedTopologies).
  2. Create a PVC + Pod; it correctly binds and schedules onto a topology-matching node (e.g. the only node with vdo-capable=true).
  3. Delete the pod only (leave the PVC in place) and recreate it without a nodeSelector.
  4. The pod can be scheduled onto a different, non-matching node. In our case it landed on a node without kvdo loaded, and NodeStageVolume correctly failed loudly (modprobe: FATAL: Module kvdo not found) rather than silently corrupting anything — but a bound PVC lacking nodeAffinity is the root cause, and a less defensive code path could fail more silently.

Confirmed via kubectl get pv <name> -o yaml: spec.nodeAffinity is absent entirely.

Suggested fix

CreateVolume should populate PersistentVolume.spec.nodeAffinity from the same AccessibleTopology/AccessibilityRequirements data it already uses to decide AllowedTopologies gating — this is standard CSI driver behavior (see the CSI spec's CreateVolumeResponse.Volume.accessible_topology, which external-provisioner uses to set PersistentVolume.spec.nodeAffinity automatically when populated). If accessible_topology is already being returned correctly, this may already work for some cases — worth checking why it isn't showing up on the PV.

Scope note

This is a pre-existing, orthogonal gap — not introduced by #402, and not blocking it (the PR works around it for its own test verification by pinning nodeSelector explicitly). Affects any StorageClass using allowedTopologies today (e.g. DHCHAP's allowedNodes gate), not just the new VDO feature.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions