Skip to content

fix: send and store real sub-second emission durations - #1374

Open
davidberenstein1957 wants to merge 2 commits into
masterfrom
fix/subsecond-emission-duration
Open

fix: send and store real sub-second emission durations#1374
davidberenstein1957 wants to merge 2 commits into
masterfrom
fix/subsecond-emission-duration

Conversation

@davidberenstein1957

@davidberenstein1957 davidberenstein1957 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Extracted from #1203 (feat/add-fastapi-middleware), which bundled this with unrelated FastAPI middleware work. It stands alone and can be reviewed and merged independently; #1203 will be rebased to drop the duplicated hunks.

What

  • duration becomes float instead of int in codecarbon/core/schemas.py and carbonserver/carbonserver/api/schemas.py (EmissionBase).
  • ApiClient.add_emission no longer refuses emissions shorter than one second, and no longer truncates the duration with int(...). It does still skip a non-positive duration: the server declares duration as Field(..., gt=0), so a zero-length flush would be a 422. Skipping rather than clamping, because clamping would invent a duration that was never measured.
  • ExperimentReport, ProjectReport and OrganizationReport widen duration from int to float.

Why

The DB column and the ORM were always Column(Float) (carbonserver/carbonserver/api/infra/database/sql_models.py) — only the pydantic models claimed int. So nothing about storage changes here; the schemas are simply being made honest about what the database already holds. On the client side the < 1 guard silently dropped any measurement shorter than a second, which is every short-lived task.

The report widening is not cosmetic: those endpoints SUM() a Float column into a field declared int. Today every stored duration happens to be a whole number, so it validates by luck. The first sub-second duration in the database makes the sum non-integral and the response model raises a validation error — a latent 500 on the experiment/project/organization report endpoints. Widening the reports is therefore part of this fix, not a separate cleanup.

No DB migration is needed. The emissions.duration column is already Column(Float) in carbonserver/carbonserver/api/infra/database/sql_models.py — this PR only changes the pydantic layer, the storage type is unchanged.

Tests

  • carbonserver/tests/api/test_schema_compatibility.py::test_millisecond_duration_survives_client_to_server — a client payload with duration=0.0042 validates against the server schema unchanged.

  • tests/test_api_call.py::TestApi::test_add_emission_sends_millisecond_duration_unchanged — replaces test_add_emission_skips_short_duration; asserts the POST body carries 0.0042.

  • tests/test_api_call.py::TestApi::test_add_emission_skips_zero_duration — a duration=0.0 flush is dropped client-side and never reaches the server's gt=0 validator.

Both fail on master (ValidationError, and emissions not sent because of a duration smaller than 1) and pass with the fix.

uv run pytest tests/ -q --ignore=tests/test_viz_data.py → 626 passed, 21 skipped. carbonserver unit tests → 108 passed. pre-commit run --all-files clean.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.43%. Comparing base (3ec31a0) to head (735be78).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1374   +/-   ##
=======================================
  Coverage   91.43%   91.43%           
=======================================
  Files          49       49           
  Lines        5057     5057           
=======================================
  Hits         4624     4624           
  Misses        433      433           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@davidberenstein1957
davidberenstein1957 marked this pull request as ready for review August 13, 2026 05:05
@davidberenstein1957
davidberenstein1957 requested a review from a team as a code owner August 13, 2026 05:05
@davidberenstein1957
davidberenstein1957 force-pushed the fix/subsecond-emission-duration branch from dfe1c2f to ed11471 Compare August 19, 2026 09:18
davidberenstein1957 and others added 2 commits August 19, 2026 16:10
The `duration` field was typed `int` in the pydantic schemas while the DB
column and ORM were always `Float`, and `ApiClient.add_emission` dropped
any measurement shorter than one second.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@davidberenstein1957
davidberenstein1957 force-pushed the fix/subsecond-emission-duration branch from ed11471 to 735be78 Compare August 19, 2026 14:12
davidberenstein1957 added a commit that referenced this pull request Aug 19, 2026
Adds an ASGI middleware that gives each HTTP request its share of a
long-running tracker's energy, plus the attribution model behind it.

One tracker runs for the app's lifetime. Each completed sampling window
(t_prev, t_now, dE) is split across the requests in flight during it,
weighted by their overlap with the window and normalised by the sum of the
weights. Windows with nothing in flight are recorded as unattributed. The
invariant attributed + unattributed == settled holds exactly after every
window, and is what the concurrency test pins down.

Why not per-request start/stop energy snapshots: with N requests in flight
each request observes the whole machine's delta, so the sum overcounts by
roughly N - measured up to 88x at 100 concurrent requests. Fair-share
weighting is the only split that conserves the run total.

A request's share is only known one or more sampling windows after its
response was sent, so results are reported then, via a callback. A request
that never covered a completed window reports energy_kwh=None rather than
zero: there is no honest number for it.

Tracker side: add_energy_window_observer / remove_energy_window_observer
expose the sampling windows, and http_request_emissions() scales the run's
EmissionsData down to one attributed share using the run's accumulated
component ratios and carbon intensity.

Depends on #1374 (duration int -> float in the emissions schemas, and
dropping the duration < 1 send guard) and #1375 (scheduler pause handling
around tasks). Both are carried by their own PRs rather than duplicated
here, so this should merge after them.

Deliberately left out, to keep the diff reviewable: hardware-tier gating of
which backends can resolve a sampling window, include/exclude path filtering
(endpoint labelling is two lines inline), idle-baseline subtraction,
per-endpoint aggregation, routing per-request rows into the tracker's own
CSV/API output handlers, a lifespan helper, and a dedicated docs page. Each
is additive on top of this and can follow if there is demand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
davidberenstein1957 added a commit that referenced this pull request Aug 19, 2026
Adds an ASGI middleware that gives each HTTP request its share of a
long-running tracker's energy, plus the attribution model behind it.

One tracker runs for the app's lifetime. Each completed sampling window
(t_prev, t_now, dE) is split across the requests in flight during it,
weighted by their overlap with the window and normalised by the sum of the
weights. Windows with nothing in flight are recorded as unattributed. The
invariant attributed + unattributed == settled holds exactly after every
window, and is what the concurrency test pins down.

Why not per-request start/stop energy snapshots: with N requests in flight
each request observes the whole machine's delta, so the sum overcounts by
roughly N - measured up to 88x at 100 concurrent requests. Fair-share
weighting is the only split that conserves the run total.

A request's share is only known one or more sampling windows after its
response was sent, so results are reported then, via a callback. A request
that never covered a completed window reports energy_kwh=None rather than
zero: there is no honest number for it.

Tracker side: add_energy_window_observer / remove_energy_window_observer
expose the sampling windows, and http_request_emissions() scales the run's
EmissionsData down to one attributed share using the run's accumulated
component ratios and carbon intensity.

Depends on #1374 (duration int -> float in the emissions schemas, and
dropping the duration < 1 send guard) and #1375 (scheduler pause handling
around tasks). Both are carried by their own PRs rather than duplicated
here, so this should merge after them.

Deliberately left out, to keep the diff reviewable: hardware-tier gating of
which backends can resolve a sampling window, include/exclude path filtering
(endpoint labelling is two lines inline), idle-baseline subtraction,
per-endpoint aggregation, routing per-request rows into the tracker's own
CSV/API output handlers, a lifespan helper, and a dedicated docs page. Each
is additive on top of this and can follow if there is demand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant