feat(prefect): add FRED and BEA API keys to both worker namespaces - #153
Open
rdahis wants to merge 4 commits into
Open
feat(prefect): add FRED and BEA API keys to both worker namespaces#153rdahis wants to merge 4 commits into
rdahis wants to merge 4 commits into
Conversation
Adds tooling for the kubeseal workflow that was previously undocumented and done by hand, plus a CLAUDE.md capturing the conventions it encodes. utils/main.py gains four commands: - seal-secret, which builds a Secret manifest in memory and pipes it into kubeseal, so plaintext never touches disk. The namespace is read from the directory's namespace.yaml and the slot number follows the highest in use. - seal-value, which adds or rotates one key inside an existing SealedSecret without needing the plaintext of its neighbours. - fetch-sealing-cert, which caches the controller's public certificate so later sealing needs no cluster access. - lint-secrets, which fails on structural mistakes and reports which snapshot is live wherever a Secret has several. The last of those documents a convention worth stating explicitly: numbered files are successive full snapshots, not fragments that merge, so only the highest-numbered file for a given metadata.name is live. Nine of the eleven files in k8s/website/django/prod are dead history.
…aces Seals a new api-keys Secret into prefect-worker-basedosdados and prefect-worker-basedosdados-dev, for the FRED and BEA data pipelines. Sealing is scoped to namespace and Secret name, so the two ciphertexts differ and neither file decrypts in the other namespace. Both work pools' base job templates were updated in the Prefect server to list api-keys in envFrom, without which the flow-run pods would never see the variables. That configuration lives in the server database, not here. Drops the leading '---' that seal-secret was adding: pretty-format-yaml strips it, so writing one guaranteed a dirty file at commit time. lint-secrets now warns about the marker's presence rather than its absence.
The certificate is encrypt-only, so this is not about confidentiality. It is about who can author a secret. This repository is public, and a SealedSecret diff is unreviewable by construction -- ciphertext in, ciphertext out. As long as producing valid ciphertext requires cluster access, the people who can seal a secret are exactly the people already trusted to apply one. Committing the certificate would break that correspondence: anyone could open a PR sealing an arbitrary value into vault-credentials, with nothing visible in review. Moves the default path to .sealed-secrets-cert.pem and gitignores it. Caching it locally is still fine and still requires cluster access to fetch, so `make fetch-sealing-cert` stays. Documents why, in CLAUDE.md and in the add-secret skill, so the convenience argument does not get re-litigated into a CI job that seals.
|
Tick the box to add this pull request to the merge queue (same as
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
FRED_API_KEYandBEA_API_KEYfor the FRED and BEA data pipelines, and automates the sealed-secret workflow that until now was undocumented and done by hand.Already applied to the cluster
The two SealedSecrets are live in
prefect-worker-basedosdadosandprefect-worker-basedosdados-dev, and both work pools' base job templates now listapi-keysinenvFrom— without that, flow-run pods would never see the variables. That template lives in the Prefect server database, not this repo, so it is not part of this diff. Flow runs pick the values up from their next run; nothing needs restarting.Scoped to prod and dev only. Pipelines has no staging worker, and
BASEDOSDADOS_CREDENTIALS_STAGINGnames a BigQuery tier rather than a deployment environment.Tooling
Four commands in
utils/main.py, wrapped asmaketargets:seal-secretnamespace.yamlseal-valuefetch-sealing-certlint-secretsTwo conventions worth knowing, now documented in CLAUDE.md
Numbered files are full snapshots, not fragments that merge. The controller keeps whichever was applied last, so only the highest-numbered file for a given
metadata.nameis live. Eight of the nineapi-prod-secretsfiles underk8s/website/django/prod/are dead history — editing one of those changes nothing while looking like it should.make lint-secretsnow prints the live file for every Secret that has more than one.pretty-format-yamlstrips the leading---, which is why most manifests lack it.seal-secretno longer writes one, since doing so guaranteed a dirty file at commit time.The sealing certificate is deliberately not committed
It is encrypt-only, so this is not a confidentiality question. It is about who can author a secret. This repository is public and a SealedSecret diff is unreviewable by construction — ciphertext in, ciphertext out. While producing valid ciphertext requires cluster access, the people who can seal a secret are exactly the people already trusted to apply one. Committing the certificate would break that correspondence and let anyone open a PR sealing an arbitrary value into, say,
vault-credentials, with nothing visible in review.Please do not add a CI job that seals. The reasoning is written into
CLAUDE.mdand theadd-secretskill so it does not get re-litigated as a convenience.Reviewing
Per CONTRIBUTING.md these manifests are reviewed by hand rather than by CI, and they are already applied — so this documents a live change rather than proposing one. Since the diff is ciphertext, verify against the cluster rather than by reading it: