Skip to content

QFw: attach the provider's own phase times to the job's execute span - #173

Merged
DougSO merged 1 commit into
mainfrom
vendor-timing-qrmi
Oct 24, 2026
Merged

DougSO merged 1 commit into
mainfrom
vendor-timing-qrmi

Conversation

@DougSO

@DougSO DougSO commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What

qfw.backend.execute is one opaque block on a real device (9.5 s via QRMI, 3.5 s via QDMI on the q20 in the 2026-10-12 runs), and the only account of how that time divides into queueing, compiling and running is the provider's own. IQM keeps a timeline per job, one timestamped event per phase from created to completed.

A new optional extension, QFW_TELEMETRY_VENDOR=1, flag-guarded like the transport spans:

  • the seconds per phase (queue, validation, compilation, execution, post_processing, total) go on the execute span as qfw.vendor.<phase>_s and into a qfw.vendor.duration histogram labelled by qfw.vendor.phase beside the backend labels, so a dashboard gets an "inside the device" row per API path and device;
  • durations only, never spans placed at the provider's timestamps, which are another clock (the doc's clocks section).

Where the data comes from

  • Native IQM QPM: it already parsed the timeline into its timing summary; the same phases now reach the telemetry.
  • QRMI path, IQM only: QRMI's task_result returns the measurements alone, and the timeline is available only as the text of task_logs. After a completed job the driver asks for it and services/util/iqm_timeline.py reads the phase durations from the text. That is one more request to the provider per job, which is why the flag. A QRMI change that returns the timeline with the result is proposed upstream as iqm: return the job's record with the task result qiskit-community/qrmi#330; when it lands the text route can go.
  • QDMI: the job interface exposes counts and a queue position and nothing with a timestamp, so that path carries no vendor timeline, and the histogram's QDMI series is absent.

Tests

tests/test_iqm_timeline.py (the parser against QRMI's exact text format, partial and empty timelines), two tests in tests/mock/test_instrumentation.py (off by default; on: attributes on the execute span, histogram points labelled by phase, api path and device), and tests/mock/test_qrmi_vendor_timing.py (flag off asks nothing of the provider; a completed IQM job is asked once; other providers are left alone; unreadable text records nothing). Unit 24 passed, the mock suite 841 passed, ci-syntax clean.

Not run against the q20: the tunnel is down today. The parser is written to QRMI's task_logs format string, and the live check follows when the tunnel is back.

qfw.backend.execute is one opaque block of seconds on a real device, and
the only account of how that time divides into queueing, compiling and
running is the provider's own: IQM keeps a timeline per job, one
timestamped event per phase. The native IQM QPM already parses it and
kept it to itself; the shim got nothing, because QRMI returns the
measurements alone from task_result and prints the timeline only as
the text of task_logs.

A new optional extension, QFW_TELEMETRY_VENDOR=1, flag-guarded like the
transport spans: the seconds per phase (queue, validation, compilation,
execution, post_processing, total) go on the execute span as
qfw.vendor.<phase>_s and into a qfw.vendor.duration histogram labelled
by phase beside the backend labels. Durations only, never spans at the
provider's timestamps, which are another clock. The native QPM records
them from its summary; the shim, for IQM only, asks task_logs after a
completed job and reads the timeline from the text (util/iqm_timeline),
one more request to the provider per job, which is why the flag. QDMI
exposes no timing, so that path carries none.

Tests: the parser against QRMI's text format, the recorder on and off
under backend_execution, and the driver's gate, provider check and
wiring with a fake resource. The design doc gains the extension.

Signed-off-by: Doug Oucharek <dougso@me.com>
@DougSO

DougSO commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Live on the ORNL q20 (2026-10-23, tunnel back): the site shim QPM run from an install of this branch with QFW_TELEMETRY_VENDOR=1, one async_run of the x circuit (10 shots) through QRMI and one through QDMI, both PASS.

QRMI path, execute span d0b53b8cdb537ea6db98d67800ba5270:

phase seconds
queue 0.034
validation 0.001
compilation 0.060
execution 1.240
post_processing 0.008
total (created to completed) 1.547

on the span as qfw.vendor.*_s, and as six qfw_vendor_duration_seconds series with qfw_stack_api_path="qrmi" in Prometheus. The execute span itself was 7.96 s: the device accounts for 1.5 s of it, and the other 6.4 s is on our side of the provider, the poll cadence and the result fetch. That gap is exactly what the extension exists to show.

QDMI path, execute span 78048fcba92f752338f05f49bbc48666: 3.80 s, no vendor attributes and no series, as designed.

The parse of task_logs' text worked first time against the real server output.

@DougSO
DougSO merged commit eacec30 into main Oct 24, 2026
10 checks passed
@DougSO
DougSO deleted the vendor-timing-qrmi branch October 24, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant