Publish local runs to analytics_dev on the real database - #1
Open
lassebenni wants to merge 1 commit into
Open
Conversation
Ports final-project-template#18: analytics_dev schema, dev publish role, table stamp on publish, and Astro no longer overrides the DB target back to Docker Postgres. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
This PR ports the “publish local runs to analytics_dev on the real database” workflow into this repo so trainees’ local pipeline runs exercise the same TLS/firewall/grant constraints as the scheduled run, without risking writes to production analytics.
Changes:
- Extend
scripts/db-setup.pyto create ananalytics_devschema +analytics_dev_userrole and grant read/write boundaries accordingly. - Stamp published backend tables with a
COMMENTdescribing the warehouse source schema + timestamp, with tests for comment placement and optionality. - Update local Airflow + docs/env examples to stop forcing publish back to Docker Postgres and to default local publishes to
analytics_dev.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/db-setup.py | Adds analytics_dev schema/role and multi-schema read-only grants. |
| data/src/publishing/sync.py | Adds optional source stamping via COMMENT ON TABLE. |
| data/tests/publishing/test_sync.py | Tests the new stamping behavior and ordering. |
| data/airflow/dags/pipeline_dag.py | Passes publish schema setting and source into publish(). |
| data/airflow/docker-compose.override.yml | Removes local-DB overrides so Astro uses the same backend DB settings as uv run. |
| data/.env.example | Updates backend DB variables to target the real DB and analytics_dev_user. |
| data/README.md | Documents new local publish target (analytics_dev) and the source-stamp behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+89
to
+90
| # looking at. | ||
| BACKEND_PG_HOST= # your teacher gives you this |
Comment on lines
+95
to
+96
| BACKEND_PG_USER=analytics_dev_user | ||
| BACKEND_PG_PASSWORD= # your teacher gives you this |
Comment on lines
+319
to
+320
| | `BACKEND_PG_PUBLISH_SCHEMA` | `analytics_dev` | `analytics` | | ||
| | `BACKEND_PG_USER` | `analytics_dev_user` | `analytics_user` | |
Comment on lines
362
to
+365
| It reads `<catalog>.dev_yourname.fct_postings_enriched` and writes | ||
| `analytics.fct_postings` in your own Postgres, the one `scripts/db-setup.py` | ||
| created. `dbt_build` runs the same way. The `ingest` task does not: it starts a | ||
| `analytics_dev.fct_postings` in the real backend database. Same table name as | ||
| production, one schema across, so promoting it later changes nothing the | ||
| backend selects. `dbt_build` runs the same way. |
| "fct_postings", | ||
| columns, | ||
| rows, | ||
| source=f"{Warehouse.from_env().catalog}.{setting('DBT_SCHEMA')}", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Local pipeline runs publish to
analytics_devon the team's real Postgres instead of a Docker copy, so TLS, firewall and grants get tested before the scheduled run.Situation
Trainees run ingest, dbt and publish on their laptops. The template still pointed publish at a local Postgres container.
Problem
A container hides the failures that show up on the first real scheduled run: TLS (
sslmode=require), the Azure firewall, and whether the database role may write the target schema.Solution
Port final-project-template#18: add
analytics_dev+analytics_dev_user, point.env.exampleat the real host, stamp published tables with the source schema, and stop Astro from overriding the publish target back to Docker.Example
A trainee sets
BACKEND_PG_USER=analytics_dev_userandBACKEND_PG_PUBLISH_SCHEMA=analytics_dev, runs publish, and rows land inanalytics_dev.fct_postingson the team VM — same table name production uses, different schema.Test plan
python -m pytest data/tests/publishing/test_sync.pyastro dev startthenairflow tasks test publish_to_backendwrites toanalytics_devscripts/db-setup.pyon team Postgres ifanalytics_dev_userdoes not exist yetanalytics_dev_userpassword via Key Vault (fp-pg-analytics-dev-team-c)Made with Cursor