Skip to content

Hourly rollup tables for gateway activity and uptime #136

Closed as not planned

Description

@bansalayush247

Problem

Gateway metrics are computed from raw history on every refresh. The activity query joins transactions → LN outputs → contract details → inputs across the whole window, and uptime and the trend scan every 5-minute poll snapshot in the window. On a large federation, 90D takes ~4 s.

#135 serves the gateway page from a precomputed in-memory overview, but the background refresh still re-runs these queries. On the shared staging/production VPS, one refresh cycle over 17 federations took ~75 s and raised CPU from ~27% to ~35%. #135 now refreshes 30D/90D less often, which helps, but the cost of each run still grows with history.

Relevant files:

  • fmo_server/src/federation/gateways.rs (list_federation_gateways, federation_gateway_uptime_trend, fetch_and_store_gateways)
  • fmo_server/src/federation/observer.rs (session processing, where LN outputs/inputs are stored)

Proposed changes

Keep small hourly rollups that are updated incrementally, and answer every window by summing them:

  • gateway_activity_hourly (federation_id, gateway_key, hour, fund_count, settle_count, cancel_count, volume_msat): updated as sessions are processed (a fund, settle or cancel attributed to a gateway adds to its current hour).
  • gateway_uptime_hourly (federation_id, gateway_id, hour, seen_samples, total_samples): updated by each gateway poll, alongside gateway_poll_snapshots.
  • Windows (24h/7d/30d/90d) become SUM over at most 2,160 rows per gateway. The daily trend is the same rows grouped by day.
  • Hourly rather than daily buckets, so the rolling 24h window stays accurate to within an hour.
  • One-time backfill migration from existing transactions / gateway_poll_snapshots.
  • With windows this cheap, the overview could be computed on request and the background refresh dropped.

The same tables would serve future per-gateway detail pages and LNv2 activity (#132).

Acceptance criteria

  • Every window answers in milliseconds, and the cost no longer grows with history.
  • Rollup results match the current queries on a production snapshot (counts, volume, seen/total samples per gateway and window), checked before switching over.
  • The retired-gateway rule (no samples counted 7 days after last_seen) still holds.
  • Backfill is resumable and safe to re-run.
  • CPU on the VPS stays at or below its level before Gateway page: precomputed overview endpoint, faster windows, stable view #135.

Activity

  1. bansalayush247 commented on Oct 11, 2026

    @bansalayush247
    MemberAuthor

    Closing for now: #137 serves the gateway page from precomputed overviews, which covers the immediate need. Rollup tables can be revisited later as part of a broader rework.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions