[SHOT-231] Add scheduled seeded database workflow - #8189
Draft
mimartin12 wants to merge 45 commits into
Draft
Conversation
… readable output when piping into other tooling
- Fix zizmor findings
Resolve ServiceCollectionExtension.cs: keep main's attachment storage, licensing, and push registrations alongside the branch's data protection key persistence. Drop the branch's stdout-redirect guard on PrintBanner — main routes the banner to stderr, so stdout is already clean for JSON piping.
The sidecar metadata file only fed two OCI labels and had already drifted — 20 of 28 presets were covered, and the rest would have shipped as category=unknown behind a warning rather than a failure. Preset names are <category>.<name>, matching their fixture folder, so the category is derivable with no file to maintain. Removes the jq dependency. Declare ARG PRESET_CATEGORY and emit bitwarden.seeder.category in each Dockerfile; the build arg was previously passed and silently discarded, so the label never reached the images. Drop the DB_TYPE build arg for the same reason — it is unused and already encoded in the image name. Export ASPNETCORE_ENVIRONMENT=Development so the seeder loads the connection strings its Azure-backed repositories require; without it GlobalSettingsFactory defaults to Production and seeding fails on a null connection string. Collapse the repeated sqlcmd invocations and three copy-paste polling loops in the MSSQL entrypoint into a wrapper plus a wait_for helper, which now fails the container on timeout instead of attaching a database that may never have come online. Rename MSSQL_PID to SQLSERVR_PID so it stops shadowing the SQL Server edition variable set in the Dockerfile. Track the seeder .gitignore that keeps data protection keys, license files, and generated seed dumps out of commits, and correct the README, which documented a _SeederMetadata table removed in b8a1ad4 and a PersistKeysToFileSystem gap that has since been fixed.
docker inspect ran the instant `docker run -d` returned, but the port binding is not always in place by then. The Go template indexed the port list unconditionally, so an empty list aborted the script outright: template parsing error: ... error calling index: reflect: slice index out of range mysql and mariadb both failed this way on a cold run; postgres and mssql only passed by winning the race. CI runners start with no cached images, so this would have hit every build. Poll for up to 30s and use `with` so an empty port list yields an empty string rather than failing the template, then fail with the container log if the port never appears.
Data protection keys and attachment blobs are read by the application, not the
database, so neither can travel inside the database image — the database only
holds attachment metadata. Without the key, logins fail against seeded data;
without the blobs, attachment metadata dangles.
Both live under /etc/bitwarden/core in a deployment, so seeding now writes them
into one tree per build, published as docker/bundles/seeded-core-{db}-{preset}-
{sha}.tar.gz and uploaded as its own CI artifact. A tarball rather than a second
image because developer setups consume these too, and unpacking anywhere is more
flexible than mounting a container.
Attachment storage is forced to local disk by blanking attachment:connectionString
— AddAttachmentStorageService prefers Azure whenever one is set, and appsettings.
Development.json sets UseDevelopmentStorage=true. The resulting layout is
{cipherId}/{attachmentId}, byte-identical to AzureAttachmentStorageService's blob
names, and metadata records ContainerName "attachments" either way, so the tree
imports into azurite unchanged. Verified against a seeded database: 18 attachment
rows, 18 blobs, 0 mismatched.
Keys are written via dataProtection:directory, which the seeder honours since
PersistKeysToFileSystem was wired up, replacing the write into $HOME. Absent
DP_KEY_XML a key is now generated into the bundle instead of being lost with the
build, so images are self-consistent by default.
Staged outside WORK_DIR so key material never enters the Docker build context,
and keyed by database type so concurrent invocations for one preset cannot
overwrite each other's bundle.
Also collapse the three container seed invocations, which differed only by
provider and connection string, into one.
CI failed with "Invalid licensing certificate". LicensingService resolves eagerly in SeederServiceScope, and outside self-hosted mode its constructor reads the X509 store for a certificate whose thumbprint is pinned to a hardcoded constant. Developer machines have that certificate; a runner does not. Local runs passed for that reason alone. Self-hosted mode loads licensing.cer and licensing_dev.cer from the embedded resources in Core instead, whose fingerprints match the pinned thumbprints, so no secret or generated certificate is involved. Generating one is not possible: the check compares against a fixed SHA-1 thumbprint. This does not change the seeded rows. Nothing under util/Seeder reads globalSettings.SelfHosted; every SelfHost value comes from preset or recipe options, and Organization.SelfHost is set per plan in PlanFeatures. Verified against a built image: SelfHost matches the Enterprise plan and LicenseKey is null. Self-hosted mode also swaps the Azure-backed event repository for a no-op, which removes the reason for exporting ASPNETCORE_ENVIRONMENT=Development, so that workaround is gone. Confirmed by seeding an attachment preset under Production: 18 attachment rows and 18 blobs in the bundle, unchanged. Also correct the bundle filename in the README, which lacked the database segment, and reword the comments added along the way.
Without a key the seeder generates one per job, so the four images in a run each got a different key and only worked with their own bundle. Worse, the stable tag moves while a downloaded bundle does not, so a consumer on that tag would hit a key mismatch and fail to log in. DP-KEY-XML now lives in the gh-org-bitwarden vault, matching how every other secret in this repo is retrieved. The build job takes the Azure login, retrieve, logout sequence and passes the key through to the script, which already prefers DP_KEY_XML over generating one.
Data Protection silently generated a key when none was supplied, producing an image whose encrypted fields opened only with that build's bundle. The build now fails instead. The README claimed images are pushed to bitwardenprod.azurecr.io, but the workflow sets PUSH=false and has no registry login, so the chart examples pointed at references that do not exist. Document downloading the run artifacts, loading the image, unpacking the bundle, and polling for the seed to finish, and note that artifacts are deleted after 7 days.
Five presets across four databases is twenty matrix jobs, so add a concurrency group to stop pushes stacking full matrices.
The core bundle unpacks to a core/ layout that matches classic self-host. BW Lite reads data-protection, attachments, and licenses directly under /etc/bitwarden, so the existing instructions put the key where lite never reads it and login fails. Note both layouts and add the steps to stage, start, and log in to lite against a seeded image.
sqlcmd returns 0 on a T-SQL error, so a failed attach still logged "Attach complete" and left a container serving no seeded data. Pass -b so set -e fires, and skip the copy and attach when vault is already present, since the data directory is usually a volume that survives a restart.
Each matrix job appended its own heading and line, so a full run left twenty blocks to scroll. Aggregate them in a final job instead.
Admin migrates on startup, so on a fresh volume it created and migrated an empty vault before the image finished attaching the seed. The attach then found the name taken and the deployment served a schema with no data. Report healthy only once the seed is attached so dependents can wait, and refuse to start when a vault exists without seeded rows rather than leaving the empty one in place.
It moved with every build exactly like the new -latest tag, so builds carried two tags meaning the same thing. Leaves one moving tag and one per-commit tag.
Contributor
🤖 Bitwarden Claude Code ReviewOverall Assessment: APPROVE Reviewed the Code Review Details
|
Seeded images stood on SQL Server 2022 while self-host ships 2025. Attach is forward-only, so the mismatch only happened to work in this direction.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8189 +/- ##
==========================================
- Coverage 63.65% 63.17% -0.48%
==========================================
Files 2366 2381 +15
Lines 102727 103771 +1044
Branches 9320 9394 +74
==========================================
+ Hits 65387 65559 +172
- Misses 35113 35981 +868
- Partials 2227 2231 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
mimartin12
changed the title
A zero-row SELECT is not a sqlcmd error, so the guard and the healthcheck both passed on a migrated-but-empty vault.
Path filter covers all four migration projects, not just Migrator — each engine migrates from a different one. Concurrency group keys on event_name because cron and push both resolve to refs/heads/main with empty inputs, so any merge cancelled the in-flight weekly build. Also bumps runners to ubuntu-24.04 and drops the Lite chart from the README.
Comment on lines
+4
to
+13
| push: | ||
| branches: [main] | ||
| paths: | ||
| - .github/workflows/build-seeded-databases.yml | ||
| - util/SeederUtility/** | ||
| - util/Seeder/** | ||
| - util/Migrator/** | ||
| - util/MsSqlMigratorUtility/** | ||
| - util/PostgresMigrations/** | ||
| - util/MySqlMigrations/** |
Contributor
There was a problem hiding this comment.
🎨 SUGGESTED: This path filter matched 116 commits in the last 3 months, so main will now fan out 20 heavy build jobs about once a day.
Details and fix
git log --since="3 months ago" -- util/Migrator util/Seeder util/SeederUtility util/PostgresMigrations util/MySqlMigrations util/MsSqlMigratorUtility returns 116 commits — migration scripts land on main most days. Each matching merge fans out to 5 presets × 4 engines = 20 jobs, and _DEFAULT_PRESETS includes scale.xl-broad-initech (10,000 users, 12,000 collections, 15,000 ciphers) plus two scale.lg-* presets, each seeded, dumped, and imaged four times. Every job then docker saves the image and uploads it with retention-days: 7, so roughly a week's worth of runs (~180 tarballs) stays resident in artifact storage.
Since PUSH is "false" here, none of that output is consumed — the value of the push trigger is the signal that seeding still works after a schema change. One small preset on one engine gives the same signal:
elif [[ "${GITHUB_EVENT_NAME}" == "push" ]]; then
presets='["qa.enterprise-basic"]'
databases='["postgres"]'keeping the full matrix on cron and dispatch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎟️ Tracking
SHOT-231
📔 Objective
Adds a scheduled workflow that builds pre-seeded database images, so an environment
can start from seeded data without running the seeder itself.
scripts/build-seeded-image.shseeds a database, snapshots it, and bakes the snapshotinto an image for postgres, mysql, mariadb, or mssql. The workflow runs a
preset × databasematrix over five presets and four engines every Sunday; manualdispatch takes a single preset and engine, or
all.MSSQL attaches MDF/LDF files instead of restoring a
.bak, since.bakrestore failson Kubernetes PVCs.
Each image gets two tags:
-latestmoves with every build,-{git-sha}is immutableso a deployment can pin one. Preset and source commit go on as labels.
Nothing pushes to the registry yet, so images come from the run's artifacts.
util/SeederUtility/scripts/README.mdshows you how to use them.