Skip to content

charts/redpanda: strip server-set metadata from looked-up bootstrap user Secret#1659

Merged
RafalKorepta merged 2 commits into
mainfrom
dyu/bootstrap-user-ssa-managed-fields
Jul 14, 2026
Merged

charts/redpanda: strip server-set metadata from looked-up bootstrap user Secret#1659
RafalKorepta merged 2 commits into
mainfrom
dyu/bootstrap-user-ssa-managed-fields

Conversation

@david-yu

@david-yu david-yu commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Problem

Helm 4 made server-side apply the default. On any helm upgrade of a release with auth.sasl.enabled=true, the chart looks up the existing *-bootstrap-user Secret (to keep the generated password stable) and re-renders it verbatim — including the server-populated metadata.managedFields. The API server rejects such apply requests, so every upgrade fails:

Error: UPGRADE FAILED: server-side apply failed for object <ns>/<release>-bootstrap-user /v1, Kind=Secret: metadata.managedFields must be nil

Reproduced on the latest published chart (26.1.8) with Helm v4.2.1: fresh install succeeds, the immediate no-op helm upgrade fails. The failure also poisons release history — any successful client-side revision stores the polluted manifest, so later server-side rollbacks to it fail the same way.

Fixes #1648.

Fix

Strip server-populated metadata from the looked-up Secret before it is stored on RenderState and re-rendered:

  • metadata.managedFields — rejected outright by the API server on apply requests.
  • metadata.resourceVersion / metadata.uid — act as optimistic-concurrency preconditions when present in an apply; re-asserting stale values causes spurious conflicts (e.g. after the Secret is ever recreated).

This mirrors what the operator's apply path already does (common-go/kube Ctl.Apply calls SetManagedFields(nil) / SetResourceVersion("") before every SSA patch), which is why operator-managed clusters never hit this — the bug is Helm-CLI-only.

Testing

  • Extended TestFetchBootstrapUser (envtest-backed) to assert the fetched Secret carries no managedFields/resourceVersion/uid. The assertions fail without the fix (envtest's apiserver populates all three on create) and pass with it.
  • TestTemplate golden suite passes unchanged (lookups return nothing in template tests).
  • New acceptance scenario (helm-chart.feature: "Bootstrap user Secret survives upgrades without server-set metadata"): installs the chart with SASL enabled, upgrades it in place, then asserts the stored release manifest re-renders the bootstrap-user Secret without server-set metadata and with the password already in use. The assertions target the stored manifest rather than the apply because the toolchain pins Helm 3, whose client-side apply tolerates the polluted re-render — the manifest is where the regression is observable today, and it is exactly what Helm 4 server-side applies. Verified both polarities against Helm 3.19.1: chart 26.1.8's stored manifest carries managedFields/uid/resourceVersion (assertions fail), the fixed chart's does not (scenario passes end-to-end via task test:acceptance).
  • Live end-to-end verification on k3d (k8s 1.33) with Helm v4.2.1 (default SSA), auth.sasl.enabled=true:
    • Steady state: install fixed chart → no-change helm upgradedeployed, and the live Secret's sole field manager is helm/Apply (real SSA, not a silent CSA fallback). This exact step fails on 26.1.8.
    • Migration/recovery: install published 26.1.8 → control upgrade on 26.1.8 fails with the issue's error → upgrade the now-failed release to the fixed chart under default SSA → deployed.
    • Bootstrap password byte-identical across every upgrade (the lookup still does its job), and the re-rendered manifest carries managedFields: null / resourceVersion: "" / uid: "", all accepted by the API server.

Backports

The same lookup path ships on all active release lines; this should be backported at least to v26.1.x (the line the issue was confirmed on) and v25.3.x.

🤖 Generated with Claude Code

…ser Secret

On upgrade the chart looks up the existing *-bootstrap-user Secret to
keep the generated password stable, and re-rendered it verbatim —
including metadata.managedFields, which the API server rejects when
Helm 4 server-side applies rendered manifests (its default):

    server-side apply failed for object <ns>/<release>-bootstrap-user
    /v1, Kind=Secret: metadata.managedFields must be nil

resourceVersion and uid are also stripped: both act as update
preconditions when present in an apply request and must not be
re-asserted from a stale read.

Reproduced on the published 26.1.8 chart with Helm v4.2.1; verified the
fixed chart upgrades cleanly under default SSA both in steady state and
when migrating a release stuck on the failure, with the bootstrap
password byte-identical throughout. The fix mirrors the sanitization the
operator's apply path already performs (common-go/kube Ctl.Apply), which
is why operator-managed clusters never hit this.

Fixes #1648

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@secpanda

secpanda commented Jul 13, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

…pgrades

Install the redpanda chart with SASL enabled, upgrade it in place, and
assert that the stored release manifest re-renders the bootstrap user
Secret without server-set metadata (managedFields, uid, resourceVersion)
and with the password already in use.

The assertions target the stored manifest rather than the apply because
the toolchain pins Helm 3, whose client-side apply tolerates the
polluted re-render — the manifest is where the regression is observable
today, and it is exactly what Helm 4 server-side applies. Without the
fix in the previous commit, the metadata assertions fail (the lookup
round-trips managedFields set by the API server); the password
assertion guards the lookup's actual job of keeping the generated
password stable across upgrades.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

💚 All backports created successfully

Status Branch Result
release/v25.3.x
release/v26.1.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

1 similar comment
@github-actions

Copy link
Copy Markdown

💚 All backports created successfully

Status Branch Result
release/v25.3.x
release/v26.1.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

@RafalKorepta RafalKorepta deleted the dyu/bootstrap-user-ssa-managed-fields branch July 14, 2026 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

helm upgrade fails under Helm 4 server-side apply: bootstrap-user Secret carries metadata.managedFields

3 participants