Skip to content

Add common stats collection objects - #610

Open
will-langdale wants to merge 1 commit into
uktrade:mainfrom
will-langdale:feat/pipe-stats
Open

Add common stats collection objects#610
will-langdale wants to merge 1 commit into
uktrade:mainfrom
will-langdale:feat/pipe-stats

Conversation

@will-langdale

@will-langdale will-langdale commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

We need a more fine-grained method to collect DAG statistics, and a common method to log new ones. This introduces it.

Fixes #419

🛠️ Changes proposed in this pull request

Adds a maximally extensible and flexible statistics object who can provide methods that are both context managers or decorators, as needed.

The way this is achieved is:

  • The DAGStats.time decorator creates a _StatLogger template. This is refreshed on each _StatLogger.__call__().
  • When _StatLogger is a decorator, name and DAGStats are inferred by introspecting self, and it logs against them. If they don't exist, it'll just print logs. Because of this it can work fine on standalone functions as well as DAGs
  • When _StatLogger is a context manager, there's no self to introspect, so you pass in the name and stats instance manually if you need 'em

👀 Guidance to review

In tests, the patching pattern is non-standard for this repo, because I think it creates a significantly more concise and readable set of tests in this instance.

update.py doesn't need the full logger object, so instead we use the adapter existing memory logging function. This function would also, one day, be perfect for _StatLogger if we wish.

🤖 AI declaration

AI written, human reviewed.

✅ Checklist:

  • This is the smallest, simplest solution to the problem
  • I've read our code standards and this code follows them
  • All new code is tested
  • I've updated all relevant documentation (select all that apply)
    • API documentation (docstrings and indexes)
    • Tutorials
    • Developer docs

@will-langdale
will-langdale force-pushed the feat/pipe-stats branch 2 times, most recently from 6019565 to eaadbd8 Compare July 7, 2026 15:13
…mory and further collection

Signed-off-by: DBT pre-commit check
@will-langdale
will-langdale marked this pull request as ready for review July 7, 2026 15:46
@will-langdale
will-langdale requested a review from a team as a code owner July 7, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Collect query statistics separate to processing statistics in DAGs

1 participant