feat(prefect): add FRED and BEA API keys to both worker namespaces

.claude/skills/add-secret/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: add-secret
+description: Add, rotate, or remove an environment variable or secret in a Kubernetes namespace in this repo — sealing it with kubeseal into a committed SealedSecret. Use whenever asked to add an API key, credential, token, password, or env var to prod, staging, or dev, for the pipelines/Prefect workers, the Django backend, the Next.js site, or the chatbot.
+---
+
+# Adding a secret
+
+Values are committed encrypted as SealedSecrets. Sealing is one command; the
+work is in choosing the right namespaces and knowing what else has to change.
+Read `CLAUDE.md` at the repository root first — it carries the conventions this
+skill assumes.
+
+## 1. Resolve the target namespaces
+
+Never guess from the words "prod" or "dev". Environment naming differs per
+application, and not every application has all three:
+
+| Application | Directories under `k8s/` |
+|---|---|
+| Prefect flows / pipelines | `prefect_workers/basedosdados` (prod), `prefect_workers/basedosdados-dev` (dev). **No staging exists** |
+| Django backend | `website/django/{prod,staging,development}` |
+| Next.js site | `website/nextjs/{production,staging,development}` |
+| Chatbot | `website/chatbot/{prod,staging}` |
+
+If the request names an environment the application does not have, say so and
+ask before inventing one — a new environment means a new namespace, Helm
+release, and (for Prefect) a new work pool, not just a secret.
+
+Confirm which Secret the value belongs in. `make lint-secrets` prints, for every
+Secret with more than one snapshot, which file is live. Existing Secrets in the
+Prefect worker namespaces are `gcp-credentials` and `vault-credentials`.
+
+## 2. Check you can seal
+
+```bash
+test -f .sealed-secrets-cert.pem && echo "cached cert, no cluster needed"
+kubectl get ns >/dev/null 2>&1 && echo "cluster reachable"
+```
+
+One of those must succeed. If neither does, stop and ask the user to run
+`gcloud auth login` — it cannot be done non-interactively.
+
+Sealing deliberately requires cluster access. Never commit
+`.sealed-secrets-cert.pem` (it is gitignored), never add a CI job that seals,
+and do not suggest either as a convenience: this repository is public and a
+SealedSecret diff is unreviewable, so requiring cluster access is what keeps
+secret authorship limited to people already trusted to apply one. `make
+fetch-sealing-cert` caches it locally, which is fine — fetching still needs
+cluster access.
+
+## 3. Seal
+
+Write the values to a file outside the repository so they never reach shell
+history or a tracked path, and delete it afterwards.
+
+**A new Secret** (a `metadata.name` not yet in that directory):
+
+```bash
+printf 'FRED_API_KEY=...\nBEA_API_KEY=...\n' > /tmp/keys.env
+make seal-secret DIR=k8s/prefect_workers/basedosdados NAME=api-keys ENVFILE=/tmp/keys.env
+make seal-secret DIR=k8s/prefect_workers/basedosdados-dev NAME=api-keys ENVFILE=/tmp/keys.env
+rm /tmp/keys.env
+```
+
+Repeat per namespace. Sealing is scoped to namespace and Secret name, so one
+namespace's file will not decrypt in another — this is not duplication that can
+be factored out.
+
+**A key added to a Secret that already exists** — target the live snapshot:
+
+```bash
+printf '...' > /tmp/value
+make seal-value FILE=k8s/prefect_workers/basedosdados/secret-04_sealed.yaml \
+  NAME=api-keys NAMESPACE=prefect-worker-basedosdados \
+  KEY=FRED_API_KEY VALUEFILE=/tmp/value
+rm /tmp/value
+```
+
+## 4. Verify
+
+```bash
+make lint-secrets
+pre-commit run --files k8s/<paths you touched>
+git diff --stat
+```
+
+The diff should touch only the intended files. Confirm the new key appears in
+`encryptedData` and that the ciphertext of untouched keys did not change.
+
+Never print a plaintext secret value back to the user, and never write one to a
+tracked file. If a value was pasted into the conversation, do not echo it in
+your summary.
+
+## 5. Say what remains
+
+Sealing writes files. It does not deploy, and for Prefect it is not even the
+whole change. Hand back an explicit list:
+
+1. **Apply** — `kubectl apply -f <each sealed file>`. Until then the controller
+   has not seen it.
+2. **Wire it up, for a new Secret name in a Prefect worker namespace** — flow-run
+   pods only receive Secrets listed in the work pool's *base job template*,
+   which lives in the Prefect server database, not this repository. Add
+   `envFrom: - secretRef: name: <secret>` at
+   `https://prefect3.basedosdados.org` → Work Pools → *pool* → Edit → Advanced,
+   once per pool. A new Secret is inert until this is done. Adding a key to a
+   Secret already listed there needs nothing.
+3. **Restart** — running pods do not pick up changed Secret values. For Prefect,
+   the next flow run gets them; other workloads need a rollout restart.
+4. **Branch and PR** — `no-commit-to-branch` blocks committing to `main`
+   directly. Kubernetes manifests are reviewed by hand, so the PR body should
+   say which namespaces are affected and what still needs applying.

.gitignore

@@ -8,10 +8,22 @@ ENV/
 env.bak/
 venv.bak/
 .DS_Store
+__pycache__/
+uv.lock
 
 # k8s
 secret.yaml
 secrets.yaml
 secret-[0-9][0-9].yaml
 encryption.key
 *.json
+
+# Sealed-secrets public certificate. Encrypt-only, so not confidential, but
+# kept out of this public repo on purpose: needing cluster access to seal is
+# what keeps "can author a secret" aligned with "can already apply one", and a
+# SealedSecret diff cannot be reviewed on sight. Fetch it per machine with
+# `make fetch-sealing-cert`.
+.sealed-secrets-cert.pem
+
+# Claude Code: skills are shared, scratch worktrees are not
+.claude/worktrees/

CLAUDE.md

@@ -0,0 +1,201 @@
+# CLAUDE.md
+
+Infrastructure for Base dos Dados: GCP resources via Terraform, cluster
+workloads via Kubernetes manifests and Helm values.
+
+## Layout
+
+| Path | Contents |
+|---|---|
+| `terraform/` | One directory per GCP module. CI plans it, CD applies it |
+| `k8s/` | One directory per namespace or application, plus its manifests |
+| `utils/main.py` | Typer CLI for operational chores (sealing secrets, base64, installs) |
+| `Makefile` | Aliases for the Terraform container and the secret workflow |
+| `.github/workflows/` | Terraform CI/CD and Infracost |
+
+Everything in `k8s/` is applied by hand with `kubectl apply` or `helm upgrade`.
+There is no CD pipeline for Kubernetes — the commands live in a comment at the
+top of each `chart/values.yaml`.
+
+## One cluster, namespaces as environments
+
+There is a single GKE cluster, `basedosdados-dev` in `us-central1-c`, despite
+the name. Production and staging are *namespaces* inside it, not separate
+clusters. One consequence matters constantly: there is one sealed-secrets
+controller and therefore **one sealing key for every environment**.
+
+Environment naming is not consistent across applications. Check the directory
+before assuming:
+
+| Application | Environments |
+|---|---|
+| `k8s/prefect_workers/` | `basedosdados` (prod), `basedosdados-dev` (dev). **No staging** |
+| `k8s/website/django/` | `prod`, `staging`, `development` |
+| `k8s/website/nextjs/` | `production`, `staging`, `development` |
+| `k8s/website/chatbot/` | `prod`, `staging` |
+
+## Secrets
+
+Secrets are committed, encrypted, as
+[SealedSecrets](https://github.com/bitnami-labs/sealed-secrets). The controller
+in the cluster holds the private key; the repository holds only ciphertext.
+Anyone can seal a value, only the cluster can open it.
+
+### Numbered files are snapshots, not fragments
+
+This trips people up, so read it before editing anything under `k8s/`.
+
+Files are named `<namespace-dir>/secret-NN_sealed.yaml`, numbered from `00`.
+Several files in one directory routinely declare the **same** `metadata.name`.
+They are not merged. Each is a complete snapshot of that Secret at a point in
+time, and the controller keeps whichever was applied last — so **the
+highest-numbered file for a given Secret name is the live one**, and the lower
+ones are dead history nobody has deleted.
+
+`k8s/website/django/prod/` holds ten sealed files. Nine declare
+`api-prod-secrets`, of which only `secret-09_sealed.yaml` is live; the tenth is
+an unrelated Secret. Editing `secret-04_sealed.yaml` would change nothing and
+look like it should. Run `make lint-secrets` — it prints the live file for
+every Secret that has more than one snapshot.
+
+Two ways to change a Secret, both used in the history:
+
+- **Add a key to the live snapshot in place** (`make seal-value`). One line
+  changes. This is the smaller, more reviewable diff — prefer it.
+- **Seal a whole new snapshot** at the next number. Needs the plaintext of
+  every existing key, so it is only worth it when most values are changing.
+
+A genuinely *new* Secret — a different `metadata.name` — takes the next free
+number, which is how `vault-credentials` came to sit at `secret-03` alongside
+`gcp-credentials`.
+
+### Other conventions
+
+- Keys inside `encryptedData` are sorted alphabetically. Roughly half the
+  repository predates this; `make lint-secrets` reports drift as a warning
+  rather than an error, so old files are not a standing failure.
+- No leading `---`, and no `creationTimestamp`. The `---` is not a style
+  preference: `pretty-format-yaml` strips it, so a file committed with one
+  comes back modified. The older files under `k8s/prefect_workers/` still have
+  theirs only because the hook has not touched them since.
+- Plaintext `secret-NN.yaml` is gitignored. Prefer never creating one — the
+  commands below stream plaintext through a pipe and never write it to disk.
+
+### Adding a new Secret
+
+```bash
+printf 'SOME_API_KEY=abc123\nOTHER_KEY=def456\n' > /tmp/new-keys.env
+make seal-secret DIR=k8s/prefect_workers/basedosdados NAME=api-keys ENVFILE=/tmp/new-keys.env
+rm /tmp/new-keys.env
+```
+
+The namespace is read from the directory's `namespace.yaml`, and the file lands
+in the next free `secret-NN` slot. Repeat per environment: each namespace needs
+its own file, even when the value is identical, because sealing is scoped to
+namespace and Secret name by default.
+
+### Adding or rotating one key
+
+Each value in a SealedSecret is encrypted independently, so a key can be added
+or replaced without knowing the plaintext of its neighbours. Point it at the
+*live* snapshot:
+
+```bash
+printf 'abc123' > /tmp/value && make seal-value \
+  FILE=k8s/prefect_workers/basedosdados/secret-04_sealed.yaml \
+  NAME=api-keys NAMESPACE=prefect-worker-basedosdados \
+  KEY=SOME_API_KEY VALUEFILE=/tmp/value && rm /tmp/value
+```
+
+This rewrites one line, which is what a well-scoped secret commit looks like —
+see `6be5d07`. `VALUE=` works too but lands in shell history; prefer
+`VALUEFILE=`.
+
+### Sealing requires cluster access, on purpose
+
+`kubeseal` encrypts with the controller's public certificate, which it fetches
+from the current `kubectl` context — so sealing needs a live `gcloud auth
+login`. That is a deliberate control, not friction to be engineered away.
+
+The certificate is encrypt-only and therefore not confidential; Bitnami
+documents publishing it as safe. The reason it stays out of this repository is
+different. **This repository is public, and a SealedSecret diff is unreviewable
+by construction** — ciphertext in, ciphertext out. While producing valid
+ciphertext requires cluster access, the set of people who can author a secret
+is exactly the set already trusted to apply one. Committing the certificate
+would break that correspondence and let anyone open a PR sealing an arbitrary
+value into, say, `vault-credentials`, with nothing visible in review.
+
+So: do not commit `.sealed-secrets-cert.pem`, and do not add a CI job that
+seals. Review a secret PR by who wrote it and what they say it contains, then
+confirm against the cluster after applying.
+
+If you would rather not re-authenticate for every seal, cache the certificate
+locally — it is gitignored, and fetching it still requires cluster access:
+
+```bash
+make fetch-sealing-cert   # writes .sealed-secrets-cert.pem, git-ignored
+```
+
+When that file exists, `seal-secret` and `seal-value` use it and skip the
+cluster. Delete it and re-fetch if the controller's key is rotated.
+
+### Applying
+
+Sealing writes a file; it does not deploy. The controller only sees it after:
+
+```bash
+kubectl apply -f k8s/prefect_workers/basedosdados/secret-04_sealed.yaml
+```
+
+Running pods do not pick up changed Secret values on their own. Restart the
+consumer, or let the next flow run pick it up.
+
+## Prefect
+
+`k8s/prefect3/` is the Prefect *server* — API, UI, and its Cloud SQL connection.
+It does not run flows.
+
+`k8s/prefect_workers/` holds the two workers that do. Each polls a work pool of
+the same name and launches one Kubernetes Job per flow run, in its own
+namespace. `basedosdados-dev` runs flows from PR branches without schedules;
+`basedosdados` runs scheduled production flows from `main`. The
+[pipelines](https://github.com/basedosdados/pipelines) repository targets them
+by pool name in `.github/scripts/deploy_flows.py`.
+
+**Flow-run pods do not automatically see the Secrets in their namespace.** The
+`envFrom` list lives in the work pool's *base job template*, which is stored in
+the Prefect server's database and edited in the UI at
+`https://prefect3.basedosdados.org` under Work Pools → *pool* → Edit → Advanced.
+A newly sealed Secret is inert until its name is added there, once per pool. Say
+so explicitly when handing off a new Secret — the manifest alone is not the
+whole change.
+
+Existing Secrets, both workers: `gcp-credentials` (`BASEDOSDADOS_CONFIG`,
+`BASEDOSDADOS_CREDENTIALS_PROD`, `BASEDOSDADOS_CREDENTIALS_STAGING`,
+`DBT_SERVICE_ACCOUNT`) and `vault-credentials` (`VAULT_ADDRESS`,
+`VAULT_TOKEN`). The `_PROD` / `_STAGING` suffix names a BigQuery dataset tier,
+not a deployment environment — that distinction has caused confusion before.
+
+`k8s/prefect_workers/basedosdados/` carries both `secret-01` and `secret-02` as
+`gcp-credentials`; `secret-02` is the live one.
+
+## Conventions
+
+- Commits follow `type(scope): description`; the history is mostly Portuguese
+  for prose and English for the summary line. Either is accepted.
+- `pre-commit` reformats YAML to two-space indent and trims whitespace. Run
+  `pre-commit run --files <paths>` before committing generated manifests.
+- `no-commit-to-branch` blocks direct commits to `main`. Branch, then PR.
+- Terraform changes are planned in CI and shown on the PR. Kubernetes manifests
+  are reviewed by hand — describe what you applied and when.
+
+## Verification
+
+`make lint-secrets` reads every sealed manifest, decrypting nothing and needing
+no cluster. It fails on the mistakes that break a deploy — a missing `template`
+stanza, a namespace that disagrees with the directory's `namespace.yaml`, a
+`metadata.name` that disagrees with the template's, an empty `encryptedData`.
+Style drift (unsorted keys, missing `---`) is reported as a warning; pass
+`--strict` to fail on it too. Its `note:` lines name the live snapshot for
+every Secret that has more than one.

Makefile

@@ -25,6 +25,47 @@ update-dev:
 	. .venv/bin/activate; \
 	poetry update;
 
+.PHONY: fetch-sealing-cert lint-secrets seal-secret seal-value
+
+# Python entrypoint for utils/main.py. Uses the project venv when its
+# dependencies are actually installed (`make create-dev`), otherwise falls back
+# to uv, which resolves typer on the fly. Probing for the venv directory is not
+# enough -- an empty .venv is a common leftover.
+UTILS := $(shell .venv/bin/python -c "import typer" 2>/dev/null \
+	&& echo ".venv/bin/python utils/main.py" \
+	|| echo "uv run --quiet --with typer python utils/main.py")
+
+# Fetch the sealed-secrets public certificate once, so later sealing needs no
+# cluster access. Requires a live `gcloud auth login`.
+fetch-sealing-cert:
+	$(UTILS) fetch-sealing-cert
+
+# Report which snapshot is live per Secret, and fail on structural mistakes.
+lint-secrets:
+	$(UTILS) lint-secrets
+
+# Create a new SealedSecret. The namespace is read from the directory's
+# namespace.yaml; the file lands in the next free secret-NN slot.
+#   make seal-secret DIR=k8s/prefect_workers/basedosdados NAME=api-keys ENVFILE=/tmp/keys.env
+seal-secret:
+	@test -n "$(DIR)" || (echo "set DIR=<namespace directory under k8s/>" && exit 1)
+	@test -n "$(NAME)" || (echo "set NAME=<metadata.name of the Secret>" && exit 1)
+	@test -n "$(ENVFILE)" || (echo "set ENVFILE=<file of KEY=VALUE lines>" && exit 1)
+	$(UTILS) seal-secret --directory $(DIR) --name $(NAME) --from-env-file $(ENVFILE) \
+		$(if $(INDEX),--index $(INDEX),)
+
+# Add or rotate a single key inside an existing SealedSecret, without needing
+# the plaintext of the other keys. Prefer VALUEFILE= over VALUE= -- the latter
+# lands in shell history.
+#   make seal-value FILE=... NAME=api-keys NAMESPACE=... KEY=FRED_API_KEY VALUEFILE=/tmp/fred
+seal-value:
+	@test -n "$(FILE)" || (echo "set FILE=<path to the sealed manifest>" && exit 1)
+	@test -n "$(NAME)" || (echo "set NAME=<metadata.name of the Secret>" && exit 1)
+	@test -n "$(NAMESPACE)" || (echo "set NAMESPACE=<target namespace>" && exit 1)
+	@test -n "$(KEY)" || (echo "set KEY=<key to add or replace>" && exit 1)
+	$(UTILS) seal-value --into $(FILE) --name $(NAME) --namespace $(NAMESPACE) --key $(KEY) \
+		$(if $(VALUEFILE),--value-file $(VALUEFILE),--value '$(VALUE)')
+
 .PHONY: docker-clean docker-down docker-force docker-logs docker-start docker-stop docker-up
 
 docker-clean: