charts/redpanda: strip server-set metadata from looked-up bootstrap user Secret#1659
Merged
Merged
Conversation
…ser Secret
On upgrade the chart looks up the existing *-bootstrap-user Secret to
keep the generated password stable, and re-rendered it verbatim —
including metadata.managedFields, which the API server rejects when
Helm 4 server-side applies rendered manifests (its default):
server-side apply failed for object <ns>/<release>-bootstrap-user
/v1, Kind=Secret: metadata.managedFields must be nil
resourceVersion and uid are also stripped: both act as update
preconditions when present in an apply request and must not be
re-asserted from a stale read.
Reproduced on the published 26.1.8 chart with Helm v4.2.1; verified the
fixed chart upgrades cleanly under default SSA both in steady state and
when migrating a release stuck on the failure, with the bootstrap
password byte-identical throughout. The fix mirrors the sanitization the
operator's apply path already performs (common-go/kube Ctl.Apply), which
is why operator-managed clusters never hit this.
Fixes #1648
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
…pgrades Install the redpanda chart with SASL enabled, upgrade it in place, and assert that the stored release manifest re-renders the bootstrap user Secret without server-set metadata (managedFields, uid, resourceVersion) and with the password already in use. The assertions target the stored manifest rather than the apply because the toolchain pins Helm 3, whose client-side apply tolerates the polluted re-render — the manifest is where the regression is observable today, and it is exactly what Helm 4 server-side applies. Without the fix in the previous commit, the metadata assertions fail (the lookup round-trips managedFields set by the API server); the password assertion guards the lookup's actual job of keeping the generated password stable across upgrades. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RafalKorepta
approved these changes
Jul 14, 2026
This was referenced Jul 14, 2026
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation and see the Github Action logs for details |
1 similar comment
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation and see the Github Action logs for details |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Helm 4 made server-side apply the default. On any
helm upgradeof a release withauth.sasl.enabled=true, the chart looks up the existing*-bootstrap-userSecret (to keep the generated password stable) and re-renders it verbatim — including the server-populatedmetadata.managedFields. The API server rejects such apply requests, so every upgrade fails:Reproduced on the latest published chart (26.1.8) with Helm v4.2.1: fresh install succeeds, the immediate no-op
helm upgradefails. The failure also poisons release history — any successful client-side revision stores the polluted manifest, so later server-side rollbacks to it fail the same way.Fixes #1648.
Fix
Strip server-populated metadata from the looked-up Secret before it is stored on
RenderStateand re-rendered:metadata.managedFields— rejected outright by the API server on apply requests.metadata.resourceVersion/metadata.uid— act as optimistic-concurrency preconditions when present in an apply; re-asserting stale values causes spurious conflicts (e.g. after the Secret is ever recreated).This mirrors what the operator's apply path already does (
common-go/kubeCtl.ApplycallsSetManagedFields(nil)/SetResourceVersion("")before every SSA patch), which is why operator-managed clusters never hit this — the bug is Helm-CLI-only.Testing
TestFetchBootstrapUser(envtest-backed) to assert the fetched Secret carries nomanagedFields/resourceVersion/uid. The assertions fail without the fix (envtest's apiserver populates all three on create) and pass with it.TestTemplategolden suite passes unchanged (lookups return nothing in template tests).helm-chart.feature: "Bootstrap user Secret survives upgrades without server-set metadata"): installs the chart with SASL enabled, upgrades it in place, then asserts the stored release manifest re-renders the bootstrap-user Secret without server-set metadata and with the password already in use. The assertions target the stored manifest rather than the apply because the toolchain pins Helm 3, whose client-side apply tolerates the polluted re-render — the manifest is where the regression is observable today, and it is exactly what Helm 4 server-side applies. Verified both polarities against Helm 3.19.1: chart 26.1.8's stored manifest carriesmanagedFields/uid/resourceVersion(assertions fail), the fixed chart's does not (scenario passes end-to-end viatask test:acceptance).auth.sasl.enabled=true:helm upgrade→deployed, and the live Secret's sole field manager ishelm/Apply(real SSA, not a silent CSA fallback). This exact step fails on 26.1.8.failedrelease to the fixed chart under default SSA →deployed.managedFields: null/resourceVersion: ""/uid: "", all accepted by the API server.Backports
The same lookup path ships on all active release lines; this should be backported at least to
v26.1.x(the line the issue was confirmed on) andv25.3.x.🤖 Generated with Claude Code