Fix public Grafana crash-loop: converge pyroscope datasource to its pinned uid - #236
Merged
Merged
Conversation
PR #190 pinned uid: pyroscope on the shared pyroscope datasource, but Grafana (12.3.1) cannot change an existing datasource's uid via provisioning: UpdateDataSource looks the row up by id AND the new uid, gets "data source not found", and the failed provisioning module crash-loops the instance. The public instance's database still had the old auto-generated uid, so its pod went down after the merge; grafana-private survived because its fresh database inserted the datasource with the pinned uid. Add deleteDatasources for pyroscope in the same provisioning file: Grafana deletes any pre-existing row (skipping the delete event when the datasource is re-created in the same pass, and skipping rows that do not exist) and re-creates it with the pinned uid, converging every instance — including a future rebuilt one — to uid pyroscope. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outage
After merging #190, the public
grafanapod entered CrashLoopBackOff:grafana-privatewas unaffected (Running, fresh database).Root cause
#190 pinned
uid: pyroscopeon the shared pyroscope datasource. Grafana 12.3.1 cannot change an existing datasource's uid through provisioning: the provisioner passes the existing row's id plus the uid from the file, andUpdateDataSourcefetches the row withWHERE id AND uid(datasource.go#L488-L496). The public database still had the old auto-generated uid (PAF2BCAF32F82481F), so the lookup returnedErrDataSourceNotFound, the provisioning module failed, and Grafana exits on provisioning failure — crash-loop. The private instance inserted the datasource fresh with the pinned uid, which is why it came up healthy.Fix
Add
deleteDatasources: [pyroscope]to the same provisioning file. Grafana's provisioner deletes any pre-existing row first and re-creates it with the pinned uid in the same pass — it explicitly skips the deletion event when the datasource is re-created (datasources.go#L249-L265) and skips rows that don't exist. Every instance (current public, current private, any future rebuild) converges touid: pyroscope, which the private pyroscope dashboard references.Recovery path
On merge, Flux updates the
grafana-datasourcesConfigMap → helm upgrade → both pods restart → public provisioning succeeds and the instance comes back. No manual cluster action needed.Validation
Rendered
core-productionthrough kustomize + envsubst +helm template(chart 10.5.15): the provisioning file carries thedeleteDatasourcesblock, all 7 shared datasources, and the pinned uid.🤖 Generated with Claude Code