Repository navigation
feat(helm): rework chart for the Cattr 4.1 runtime #401
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 14 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
7955bde
fix(runtime): stop baking REVERB_APP_SECRET and DB credentials into t…
vermorag 18df74d
fix(helm): never generate APP_KEY randomly and add REVERB_APP_SECRET
vermorag 8fa7461
fix(runtime): scope the REVERB_APP_KEY Trivy ignore to its own ENV in…
vermorag 91b2a21
ci: drop Dockerfile baseline ignores now that no credentials are bake…
vermorag 94cb3bb
feat(octane): configure Swoole worker counts via OCTANE_WORKERS / OCT…
vermorag 657e768
feat(runtime): CATTR_RESTART_POLICY to let the orchestrator restart c…
vermorag 9399450
feat(helm): rework the chart for the Cattr 4.1 runtime (chart 0.2.0-a…
vermorag 1325158
ci(helm): chart check job and Trivy coverage for chart 0.2
vermorag 098aab6
docs(helm): chart README and 0.1 -> 0.2 migration guide
vermorag 7dfd1cb
ci(helm): fail the lint step on template fail
vermorag 1015f9d
fix(helm): keep the setup Job off the chart ServiceAccount
vermorag 17e5a37
fix(helm): fail the render on an Ingress without hosts
vermorag 569f80e
docs(helm): rollback to 0.1.x needs the pre-upgrade dump
vermorag 142df1c
docs(octane): document OCTANE_WORKERS precedence over --workers
vermorag c0d7a03
docs(helm): take the rollback PVC size and class from the original claim
vermorag 4dcab1b
feat(helm): target the 4.1.0-alpha.2 image (chart 0.2.0-alpha.2)
vermorag 20f21af
docs(helm): post-install setup failures and binlog requirement for ex…
vermorag c3fbe1e
docs(helm): provisioning failures with setup.hook=false
vermorag 95559fc
docs(helm): separate failed pre-install and pre-upgrade setup
vermorag File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # Not part of the packaged chart. | ||
| ci/ | ||
| .git/ | ||
| *.swp | ||
| *.bak | ||
| *.tmp |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,227 @@ | ||
| # Migrating from chart 0.1.x to 0.2.x | ||
|
|
||
| Chart 0.2 is a rewrite for the Cattr 4.1 runtime. An in-place `helm upgrade` | ||
| of the same release works, but **files are not migrated automatically** and | ||
| several values changed. Plan a short downtime. | ||
|
|
||
| ## What changes | ||
|
|
||
| | | 0.1.x | 0.2.x | | ||
| |---|---|---| | ||
| | Image | `registry.git.amazingcat.net/cattr/core/app:latest` | `ghcr.io/cattr-app/server:<appVersion>` | | ||
| | Workload | one container, role `all` (migrations in every Pod) | `web`, `queue`, `scheduler` containers + setup Job | | ||
| | `replicaCount` | yes | removed (pvc mode is single-replica) | | ||
| | Files | PVCs `<fullname>-screenshots` / `-attachments` mounted at `storage/app/uploads/*` | one PVC `<fullname>-storage` with the paths the app actually uses | | ||
| | ConfigMap `<fullname>` | yes | removed (env is inline) | | ||
| | `APP_KEY` | generated randomly when unset | never generated; kept from the existing Secret | | ||
| | `REVERB_APP_SECRET` | baked into the image | required value (`cattr.reverbAppSecret`) | | ||
| | DB passwords | defaults `cattr` / `cattrroot` in values | no defaults; kept from the existing Secret on upgrade | | ||
| | Built-in MySQL | on by default, `mysql:8.0`, no PVC protection | opt-in demo (`mysql.enabled: true`), `mysql:8.4.13`, non-root, PVC kept | | ||
| | Ingress | on by default, `ingress.hostname` | off by default, `ingress.hosts[]` | | ||
| | Values | free-form | `values.schema.json` rejects unknown (0.1.x) keys | | ||
|
|
||
| **Chart 0.1.x never persisted uploaded files.** Its PVCs were mounted at | ||
| `storage/app/uploads/screenshots` and `storage/app/uploads/attachments`, but | ||
| Cattr writes `storage/app/screenshots` and `storage/app/attachments`, which | ||
| lived only in the running Pod. They are lost whenever that Pod is replaced — | ||
| so copy them out of the **running** Pod before upgrading, and do not restart | ||
| it before that. The 0.1.x PVCs are deleted by the upgrade; they contain no | ||
| application data (check with the command in step 2). | ||
|
|
||
| ## 1. Back up the database | ||
|
|
||
| The dump is mandatory: it is the only way back to 0.1.x (see | ||
| [Rollback](#rollback)). The setup Job migrates the schema, and with the | ||
| built-in MySQL the upgrade moves the data directory from MySQL 8.0 to 8.4, | ||
| which MySQL 8.0 cannot open again (there is no in-place downgrade). | ||
|
|
||
| Built-in MySQL: | ||
|
|
||
| ```sh | ||
| NS=cattr; REL=cattr; FULL=cattr-cattr-server # adjust: <release>-cattr-server | ||
| kubectl -n $NS exec deploy/$FULL-mysql -- sh -c \ | ||
| 'mysqldump -uroot -p"$MYSQL_ROOT_PASSWORD" --single-transaction --routines --triggers "$MYSQL_DATABASE"' \ | ||
| > cattr-backup.sql | ||
| ``` | ||
|
|
||
| External database: use your usual backup. Also save the Secret, it holds | ||
| `APP_KEY`: | ||
|
|
||
| ```sh | ||
| kubectl -n $NS get secret $FULL -o yaml > cattr-secret-backup.yaml | ||
| ``` | ||
|
|
||
| ## 2. Copy the files out of the running Pod | ||
|
|
||
| ```sh | ||
| POD=$(kubectl -n $NS get pod -l app.kubernetes.io/instance=$REL,app.kubernetes.io/component=app -o name | head -1) | ||
| kubectl -n $NS exec $POD -- sh -c 'cd /opt/cattr/app/storage/app && find attachments screenshots -type f | wc -l; find uploads -type f | wc -l' | ||
| mkdir -p cattr-files | ||
| kubectl -n $NS cp ${POD#pod/}:/opt/cattr/app/storage/app/attachments cattr-files/attachments | ||
| kubectl -n $NS cp ${POD#pod/}:/opt/cattr/app/storage/app/screenshots cattr-files/screenshots | ||
| find cattr-files -type f | wc -l # must match the first count | ||
| ``` | ||
|
|
||
| ## 3. Protect the MySQL volume (built-in MySQL only) | ||
|
|
||
| 0.1.x did not mark the MySQL PVC as kept. Annotate it before upgrading so a | ||
| mistake cannot delete the database: | ||
|
|
||
| ```sh | ||
| kubectl -n $NS annotate pvc $FULL-mysql helm.sh/resource-policy=keep --overwrite | ||
| ``` | ||
|
|
||
| ## 4. Switch the Deployment strategy (Helm 4 / server-side apply) | ||
|
|
||
| 0.2 uses `strategy: Recreate`. With client-side apply (Helm 3, or Helm 4 | ||
| for releases installed by Helm 3) the chart clears the old `rollingUpdate` | ||
| itself. With server-side apply (releases installed by Helm 4) the upgrade | ||
| fails with `spec.strategy.rollingUpdate: Forbidden`; switch the strategy first | ||
| (this does not restart the Pod): | ||
|
|
||
| ```sh | ||
| kubectl -n $NS patch deployment $FULL --type=json \ | ||
| -p='[{"op":"replace","path":"/spec/strategy","value":{"type":"Recreate"}}]' | ||
| ``` | ||
|
|
||
| ## 5. Upgrade | ||
|
|
||
| Write a new values file (do not use `--reuse-values`). APP_KEY and the DB | ||
| credentials are read from the existing Secret when left empty: | ||
|
|
||
| ```yaml | ||
| cattr: | ||
| appUrl: https://cattr.example.com | ||
| reverbAppSecret: "<openssl rand -hex 32>" # new, required once | ||
| # Built-in MySQL: keep it explicitly (default is now off) | ||
| mysql: | ||
| enabled: true | ||
| # ...or an external database: | ||
| # database: | ||
| # host: mysql.example.internal | ||
| setup: | ||
| admin: | ||
| enabled: false # the administrator already exists | ||
| ingress: # if you used the 0.1.x Ingress | ||
| enabled: true | ||
| className: nginx | ||
| hosts: | ||
| - host: cattr.example.com | ||
| paths: [{path: /, pathType: Prefix}] | ||
| ``` | ||
|
|
||
| ```sh | ||
| helm upgrade $REL ./.helm -n $NS -f values-0.2.yaml --wait --timeout 15m | ||
| ``` | ||
|
|
||
| The `pre-upgrade` setup Job runs the migrations; then the Pod is recreated. | ||
| With the built-in MySQL the image changes from `mysql:8.0` to `mysql:8.4.13` | ||
| (in-place upgrade of the data directory, done by MySQL on first start) and it | ||
| now runs as UID 999 without root. | ||
|
|
||
| If the render fails with "the release still runs the built-in MySQL | ||
| Deployment", you left `mysql.enabled` at its new default `false`: set it to | ||
| `true`, or migrate the dump to an external database and set | ||
| `mysql.acknowledgeRemoval: true`. | ||
|
|
||
| ## 6. Restore the files | ||
|
|
||
| ```sh | ||
| POD=$(kubectl -n $NS get pod -l app.kubernetes.io/instance=$REL,app.kubernetes.io/component=app -o name | head -1) | ||
| kubectl -n $NS cp cattr-files/attachments ${POD#pod/}:/opt/cattr/app/storage/app/ -c web | ||
| kubectl -n $NS cp cattr-files/screenshots ${POD#pod/}:/opt/cattr/app/storage/app/ -c web | ||
| kubectl -n $NS exec ${POD#pod/} -c web -- sh -c 'cd /opt/cattr/app/storage/app && find attachments screenshots -type f | wc -l' | ||
| ``` | ||
|
|
||
| Files are written as UID 10000 onto the new PVC `<fullname>-storage`. | ||
|
|
||
| ## 7. Verify | ||
|
|
||
| - `kubectl -n $NS get secret $FULL -o jsonpath='{.data.APP_KEY}'` equals the backup. | ||
| - `kubectl -n $NS logs job/$FULL-setup -c setup` ends with `DONE` (no `FAIL`). | ||
| - Log in with an existing user; open a task attachment and a screenshot. | ||
| - `kubectl -n $NS get pod` shows `3/3` containers; queue jobs are processed | ||
| (e.g. attachment upload moves from processing to available). | ||
|
|
||
| ## Rollback | ||
|
|
||
| `helm rollback` alone does not return to a working 0.1.x: | ||
|
|
||
| - **Built-in MySQL:** the upgrade moved the kept PVC `<fullname>-mysql` to | ||
| MySQL 8.4. A rollback starts `mysql:8.0` on that 8.4 data directory, which | ||
| MySQL does not support (no in-place downgrade): the server refuses to start | ||
| ("Invalid MySQL server downgrade"). The only way back is to restore the | ||
| pre-upgrade `mysqldump` from step 1 into a **fresh** MySQL 8.0 data | ||
| directory on a new, empty PVC. | ||
| - **External database:** the setup Job ran the 0.2 migrations, so restore the | ||
| database backup from step 1 before 0.1.x runs against it. | ||
| - Neither path restores the deleted 0.1.x PVCs (they held no application | ||
| data) or the files on `<fullname>-storage`; copy the files from step 2 back | ||
| into the 0.1.x Pod. | ||
|
|
||
| Built-in MySQL, step by step. You need the 0.1.x chart (e.g. checked out from | ||
| the 0.1.x release) as `./chart-0.1`, your 0.1.x values file | ||
| `values-0.1.yaml` (with the same `database.*` credentials as before) and the | ||
| backups from step 1. | ||
|
|
||
| 1. Create an empty PVC for the 8.0 data directory (same size and | ||
| StorageClass as `<fullname>-mysql`). Leave `<fullname>-mysql` untouched: | ||
| it keeps the 8.4 data in case you go forward again. | ||
|
|
||
| ```sh | ||
| kubectl -n $NS apply -f - <<EOF | ||
| apiVersion: v1 | ||
| kind: PersistentVolumeClaim | ||
| metadata: | ||
| name: $FULL-mysql-80 | ||
| spec: | ||
| accessModes: [ReadWriteOnce] | ||
| resources: | ||
| requests: | ||
| storage: 10Gi | ||
| EOF | ||
| ``` | ||
|
|
||
| 2. Deploy 0.1.x on that PVC with the application scaled to zero, so it does | ||
| not migrate or seed the empty database before the restore. 0.1.x generates | ||
| a new random `APP_KEY` when `cattr.appKey` is empty: pass the key from the | ||
| Secret backup (0.1.x expects it without the `base64:` prefix). | ||
|
|
||
| ```sh | ||
| APP_KEY=$(awk '$1 == "APP_KEY:" {print $2}' cattr-secret-backup.yaml | base64 -d) | ||
| helm upgrade $REL ./chart-0.1 -n $NS -f values-0.1.yaml \ | ||
| --set mysql.persistence.existingClaim=$FULL-mysql-80 \ | ||
| --set cattr.appKey="${APP_KEY#base64:}" \ | ||
| --set replicaCount=0 --wait --timeout 15m | ||
| ``` | ||
|
|
||
| (Use `helm upgrade` with the 0.1.x chart, not `helm rollback`: a rollback | ||
| reuses the old values, which point MySQL at the 8.4 PVC.) | ||
|
|
||
| 3. Wait until MySQL 8.0 has initialised the empty data directory, then load | ||
| the dump: | ||
|
|
||
| ```sh | ||
| kubectl -n $NS rollout status deploy/$FULL-mysql | ||
| kubectl -n $NS exec -i deploy/$FULL-mysql -- sh -c \ | ||
| 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE"' < cattr-backup.sql | ||
| ``` | ||
|
|
||
| 4. Start the application again (same command with your usual | ||
| `replicaCount`, default 1) and copy the files from step 2 back into the | ||
| Pod: | ||
|
|
||
| ```sh | ||
| helm upgrade $REL ./chart-0.1 -n $NS -f values-0.1.yaml \ | ||
| --set mysql.persistence.existingClaim=$FULL-mysql-80 \ | ||
| --set cattr.appKey="${APP_KEY#base64:}" \ | ||
| --set replicaCount=1 --wait --timeout 15m | ||
| ``` | ||
|
|
||
| 5. Once 0.1.x works, `<fullname>-mysql` (8.4 data) and `<fullname>-storage` | ||
| can be deleted by hand; both are kept by Helm. | ||
|
|
||
| External database: restore the backup from step 1 into the database (drop and | ||
| recreate it, or restore into a new database and point `database.*` at it), | ||
| then deploy 0.1.x with `values-0.1.yaml` and the `APP_KEY` from the Secret | ||
| backup as above, and copy the files back. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.