From 978f75590757f46a7117e28cc0ffc4143ab4dd09 Mon Sep 17 00:00:00 2001 From: Daryl White Date: Tue, 14 Jul 2026 15:37:18 -0400 Subject: [PATCH 1/3] docs(DOC-1618): restore k8s compatibility known issue and add troubleshooting page PR #2228 accidentally dropped the known-issue warning for 1.34+ control plane clusters paired with older tenant clusters. Restores it for the combinations affected by the pod Ready=False sync bug (ENGNODE-105 / vcluster#3578) and adds a troubleshooting page documenting symptoms, cause, and fix status. Co-Authored-By: Claude Sonnet 5 --- static/api/k8s-compatibility.json | 38 ++++++++++--- .../pod-stuck-ready-false-version-skew.mdx | 53 +++++++++++++++++++ 2 files changed, 84 insertions(+), 7 deletions(-) create mode 100644 vcluster/troubleshoot/pod-stuck-ready-false-version-skew.mdx diff --git a/static/api/k8s-compatibility.json b/static/api/k8s-compatibility.json index 4267c87f12..38324409d3 100644 --- a/static/api/k8s-compatibility.json +++ b/static/api/k8s-compatibility.json @@ -1,5 +1,5 @@ { - "lastUpdated": "2026-05-28", + "lastUpdated": "2026-07-14", "kubernetesVersions": [ "1.36", "1.35", @@ -11,9 +11,18 @@ "host": "1.36", "vcluster": { "1.36": "tested", - "1.35": "compatible", - "1.34": "compatible", - "1.33": "compatible" + "1.35": { + "status": "known-issue", + "note": 1 + }, + "1.34": { + "status": "known-issue", + "note": 1 + }, + "1.33": { + "status": "known-issue", + "note": 1 + } } }, { @@ -21,8 +30,14 @@ "vcluster": { "1.36": "compatible", "1.35": "tested", - "1.34": "compatible", - "1.33": "compatible" + "1.34": { + "status": "known-issue", + "note": 1 + }, + "1.33": { + "status": "known-issue", + "note": 1 + } } }, { @@ -31,7 +46,10 @@ "1.36": "compatible", "1.35": "compatible", "1.34": "tested", - "1.33": "compatible" + "1.33": { + "status": "known-issue", + "note": 1 + } } }, { @@ -44,6 +62,12 @@ } } ], + "notes": [ + { + "id": 1, + "text": "Pods can get stuck with a `Ready=False` status when the tenant cluster runs an older Kubernetes version than the control plane cluster. This keeps their Deployments and StatefulSets from becoming ready. See [troubleshooting steps and fix status](/docs/vcluster/troubleshoot/pod-stuck-ready-false-version-skew) for details." + } + ], "statuses": { "tested": { "emoji": "✅", diff --git a/vcluster/troubleshoot/pod-stuck-ready-false-version-skew.mdx b/vcluster/troubleshoot/pod-stuck-ready-false-version-skew.mdx new file mode 100644 index 0000000000..ef7f98b560 --- /dev/null +++ b/vcluster/troubleshoot/pod-stuck-ready-false-version-skew.mdx @@ -0,0 +1,53 @@ +--- +title: Resolve pods stuck at Ready=False with control plane and tenant version skew +sidebar_label: Pod stuck Ready=False (version skew) +description: Troubleshoot pods and Deployments stuck at Ready=False when the tenant cluster runs an older Kubernetes version than the control plane cluster. +keywords: [ready false, qosClass immutable, observedGeneration, version skew, pod sync] +--- + +When the tenant cluster runs an older Kubernetes version than the control plane cluster, synced pods can get stuck reporting `Ready=False` even though the pod itself is running. Deployments and StatefulSets never show as ready, because they wait on that pod condition. + +## Symptoms + +A pod's containers are all `Running`, but its owning Deployment or StatefulSet never reaches the desired ready count: + +```bash title="Pod is Running, but the Deployment is not Ready" +kubectl get pods +# NAME READY STATUS RESTARTS AGE +# my-app-7cf4ddcbbf-k5m2n 1/1 Running 0 127m + +kubectl get deployments +# NAME READY UP-TO-DATE AVAILABLE AGE +# my-app 0/1 1 0 127m +``` + +The syncer logs a reconcile error for the pod, similar to: + +```text title="Syncer log" +ERROR controller/controller.go:474 Reconciler error {"component": "vcluster", "controller": "pod", ..., "error": "sync: patch host object: update object status: Pod \"my-app-7cf4ddcbbf-k5m2n--b21ec34921\" is invalid: status.qosClass: Invalid value: \"BestEffort\": field is immutable"} +``` + +Restarting the control plane's API server pod can temporarily clear some of these errors, but they come back after creating or updating deployments. + +## Cause + +This is a known issue tracked in [vCluster GitHub Issue #3578](https://github.com/loft-sh/vcluster/issues/3578), with two contributing root causes: + +- **QoS class immutability**: the pod syncer copies the tenant pod's QoS class onto the host object before patching. Kubernetes 1.32 and later rejects that patch because `status.qosClass` is immutable once set. +- **`observedGeneration` mismatch**: starting in Kubernetes 1.34, the host kubelet sets `observedGeneration` on pod status and conditions. A tenant API server on Kubernetes older than 1.34 drops that field on write, since `PodObservedGenerationTracking` is alpha or off by default there. The syncer then sees the object cache and the informer disagree on every reconcile, reads that as a real condition change, and keeps overwriting the tenant's `Ready=True` condition with stale data. + +Both root causes only trigger when the tenant runs an older Kubernetes version than the control plane cluster. See the [Kubernetes compatibility matrix](/docs/vcluster/manage/upgrade/supported_versions#kubernetes-compatibility-matrix) for the specific control plane and tenant version combinations flagged with this known issue. + +## Workaround + +There is no reliable workaround available today. Restarting the control plane's API server pod clears the immediate errors, but they recur as soon as new pods sync. The most effective mitigation is to keep the tenant cluster on the same Kubernetes minor version as the control plane cluster until the fix ships. + +## Fix status + +A fix is in progress in [GitHub Issue #4037](https://github.com/loft-sh/vcluster/pull/4037), which stops the syncer from writing the immutable QoS class to the host and ignores `observedGeneration` when the tenant cluster doesn't persist it. + +## Related resources + +- [Kubernetes compatibility matrix](/docs/vcluster/manage/upgrade/supported_versions#kubernetes-compatibility-matrix) +- [vcluster#3578](https://github.com/loft-sh/vcluster/issues/3578) +- [vcluster#4037](https://github.com/loft-sh/vcluster/pull/4037) From 68cbde88e58a799c253e6fd6ad87747305e67d88 Mon Sep 17 00:00:00 2001 From: Daryl White Date: Wed, 15 Jul 2026 09:24:08 -0400 Subject: [PATCH 2/3] fix(DOC-1618): scope known-issue to 1.33 tenant only The observedGeneration bug (ENGNODE-105 / vcluster#3578) is gated on tenant version < 1.34, not simply "older than host". Narrow the known-issue marker to the tenant=1.33 column across the 1.34/1.35/1.36 host rows, matching engineering's confirmed compatibility table. Co-Authored-By: Claude Sonnet 5 --- static/api/k8s-compatibility.json | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/static/api/k8s-compatibility.json b/static/api/k8s-compatibility.json index 38324409d3..4f8b8cea22 100644 --- a/static/api/k8s-compatibility.json +++ b/static/api/k8s-compatibility.json @@ -1,5 +1,5 @@ { - "lastUpdated": "2026-07-14", + "lastUpdated": "2026-07-15", "kubernetesVersions": [ "1.36", "1.35", @@ -11,14 +11,8 @@ "host": "1.36", "vcluster": { "1.36": "tested", - "1.35": { - "status": "known-issue", - "note": 1 - }, - "1.34": { - "status": "known-issue", - "note": 1 - }, + "1.35": "compatible", + "1.34": "compatible", "1.33": { "status": "known-issue", "note": 1 @@ -30,10 +24,7 @@ "vcluster": { "1.36": "compatible", "1.35": "tested", - "1.34": { - "status": "known-issue", - "note": 1 - }, + "1.34": "compatible", "1.33": { "status": "known-issue", "note": 1 From 618526af2655a2bbff1a1d3a7530d8c383e92ea2 Mon Sep 17 00:00:00 2001 From: Daryl White Date: Wed, 15 Jul 2026 09:29:55 -0400 Subject: [PATCH 3/3] fix(DOC-1618): anchor known-issue wording to the actual version gate The note and troubleshooting page said the bug hits any tenant older than the control plane, which overstates the actual gate confirmed in vcluster PR #4037: tenant on 1.33 or earlier with control plane on 1.34 or later. Tighten the wording accordingly. Co-Authored-By: Claude Sonnet 5 --- static/api/k8s-compatibility.json | 2 +- .../troubleshoot/pod-stuck-ready-false-version-skew.mdx | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/static/api/k8s-compatibility.json b/static/api/k8s-compatibility.json index 4f8b8cea22..fb98ee2549 100644 --- a/static/api/k8s-compatibility.json +++ b/static/api/k8s-compatibility.json @@ -56,7 +56,7 @@ "notes": [ { "id": 1, - "text": "Pods can get stuck with a `Ready=False` status when the tenant cluster runs an older Kubernetes version than the control plane cluster. This keeps their Deployments and StatefulSets from becoming ready. See [troubleshooting steps and fix status](/docs/vcluster/troubleshoot/pod-stuck-ready-false-version-skew) for details." + "text": "Pods can get stuck with a `Ready=False` status when the tenant cluster runs Kubernetes 1.33 or earlier and the control plane cluster runs Kubernetes 1.34 or later. This keeps their Deployments and StatefulSets from becoming ready. See [troubleshooting steps and fix status](/docs/vcluster/troubleshoot/pod-stuck-ready-false-version-skew) for details." } ], "statuses": { diff --git a/vcluster/troubleshoot/pod-stuck-ready-false-version-skew.mdx b/vcluster/troubleshoot/pod-stuck-ready-false-version-skew.mdx index ef7f98b560..61c896b5ab 100644 --- a/vcluster/troubleshoot/pod-stuck-ready-false-version-skew.mdx +++ b/vcluster/troubleshoot/pod-stuck-ready-false-version-skew.mdx @@ -1,11 +1,11 @@ --- title: Resolve pods stuck at Ready=False with control plane and tenant version skew sidebar_label: Pod stuck Ready=False (version skew) -description: Troubleshoot pods and Deployments stuck at Ready=False when the tenant cluster runs an older Kubernetes version than the control plane cluster. +description: Troubleshoot pods and Deployments stuck at Ready=False when the tenant cluster runs Kubernetes 1.33 or earlier and the control plane cluster runs 1.34 or later. keywords: [ready false, qosClass immutable, observedGeneration, version skew, pod sync] --- -When the tenant cluster runs an older Kubernetes version than the control plane cluster, synced pods can get stuck reporting `Ready=False` even though the pod itself is running. Deployments and StatefulSets never show as ready, because they wait on that pod condition. +When the tenant cluster runs Kubernetes 1.33 or earlier and the control plane cluster runs Kubernetes 1.34 or later, synced pods can get stuck reporting `Ready=False` even though the pod itself is running. Deployments and StatefulSets never show as ready, because they wait on that pod condition. ## Symptoms @@ -36,11 +36,11 @@ This is a known issue tracked in [vCluster GitHub Issue #3578](https://github.co - **QoS class immutability**: the pod syncer copies the tenant pod's QoS class onto the host object before patching. Kubernetes 1.32 and later rejects that patch because `status.qosClass` is immutable once set. - **`observedGeneration` mismatch**: starting in Kubernetes 1.34, the host kubelet sets `observedGeneration` on pod status and conditions. A tenant API server on Kubernetes older than 1.34 drops that field on write, since `PodObservedGenerationTracking` is alpha or off by default there. The syncer then sees the object cache and the informer disagree on every reconcile, reads that as a real condition change, and keeps overwriting the tenant's `Ready=True` condition with stale data. -Both root causes only trigger when the tenant runs an older Kubernetes version than the control plane cluster. See the [Kubernetes compatibility matrix](/docs/vcluster/manage/upgrade/supported_versions#kubernetes-compatibility-matrix) for the specific control plane and tenant version combinations flagged with this known issue. +Both root causes only trigger when the tenant cluster runs Kubernetes 1.33 or earlier and the control plane cluster runs Kubernetes 1.34 or later. See the [Kubernetes compatibility matrix](/docs/vcluster/manage/upgrade/supported_versions#kubernetes-compatibility-matrix) for the specific version combinations flagged with this known issue. ## Workaround -There is no reliable workaround available today. Restarting the control plane's API server pod clears the immediate errors, but they recur as soon as new pods sync. The most effective mitigation is to keep the tenant cluster on the same Kubernetes minor version as the control plane cluster until the fix ships. +There is no reliable workaround available today. Restarting the control plane's API server pod clears the immediate errors, but they recur as soon as new pods sync. The most effective mitigation is to keep the tenant cluster on Kubernetes 1.34 or later whenever the control plane cluster runs Kubernetes 1.34 or later, until the fix ships. ## Fix status