mobile-config: serve gateways from dbt tables#1221
Draft
bbalser wants to merge 2 commits into
Draft
Conversation
Repoint the gateway service to read from the dbt-maintained chain tables (dbt.mobile_gateway_inventory + dbt.mobile_hotspot_history) instead of the locally-ingested `gateways` table, and retire the S3 change-report ingestion. - db.rs: all current-state reads now hit dbt.mobile_gateway_inventory joined to a local deployment_info cache; info_at_timestamp reads dbt.mobile_hotspot_history (latest version <= query_time). device_type is parsed from the screaming-snake dbt form and location from asserted_hex. - Delete hotspot_change_stream and ownership_change_stream daemons and their S3 pollers; drop the `gateways` table. - tracker.rs -> DeploymentInfoTracker: the only field the dbt chain tables can't supply is the WiFi (antenna, elevation) pair, so a periodic job bulk- copies it from the metadata DB into a local `deployment_info` table that reads join against, keeping the metadata DB off the request path. - settings: drop `ingest` and `gateway_stream_start_after`; add `deployment_info_interval`. - Rework the gateway integration tests to seed the dbt-shaped tables; add unit coverage for hex/device-type parsing. antenna/elevation are refresh-interval stale and current-only for info_at_timestamp (the metadata DB has no history for them) -- unchanged from prior behavior. The metadata pool remains required for the entity, carrier, and sub_dao services regardless.
Don't drop the legacy gateways table in the cutover migration so we can roll back to the old ingestion code path without data loss. A follow-up migration can drop it after the dbt-backed path has soaked.
bbalser
marked this pull request as draft
July 14, 2026 13:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Repoints the mobile-config gateway service to read from the dbt-maintained chain tables instead of the locally-ingested
gatewaystable, and retires the S3 change-report ingestion entirely.dbt.mobile_gateway_inventory(one row per gateway, current state) joined to a localdeployment_infocache.info_at_timestampreadsdbt.mobile_hotspot_history(latest version<= query_time).hotspot_change_streamandownership_change_streamdaemons + their S3 pollers. Thegatewaystable is left in place (not dropped) so we can roll back to the old code path without data loss — a follow-up migration drops it after soak.tracker.rs→DeploymentInfoTracker: the only gateway field the dbt chain tables can't supply is the WiFi(antenna, elevation)pair (not on-chain; onlyazimuthis). A periodic job bulk-copies it from the metadata DB into a localdeployment_infotable that reads join against — keeping the metadata DB off the request path.ingestandgateway_stream_start_after; addeddeployment_info_interval(default 1h).Why
The dbt models (
network-dbt) now maintain gateway state and full change history in Postgres, so mobile-config no longer needs to poll S3 and reconstruct thegatewaystable itself. mobile-config no longer touches S3.Reviewer notes
network-dbtand must be deployed for this to run:mobile_config.dbt.mobile_gateway_inventoryandmobile_config.dbt.mobile_hotspot_history.gatewaystable is retained; reverting the code restores the old path with data intact.ingest/gateway_stream_start_afterfrom the mobile-config config.antenna/elevationare refresh-interval stale and current-only forinfo_at_timestamp(the metadata DB holds no history for them) — this matches prior behavior, not a regression.metadata_poolstays required regardless: entity, carrier, and sub_dao services still depend on the metadata DB; within the gateway path only the tracker uses it now.Testing
DATABASE_URL=… cargo nextest run -p mobile-config -j 1): reworked gateway_db / gateway_service{,_v3,_v4} / gateway_tracker integration tests + new unit tests for hex-location and device-type parsing.cargo clippy -p mobile-config --all-targetsclean.