[DougSO]

What

The shim QPM now copes with a provider that rate-limits it with HTTP 429, so one 429 no longer fails a workflow.

• The QRMI driver waits a 429 out. task_start, task_status and task_result are each tried again after a 429. The wait doubles from 2 to 60 seconds, with jitter drawn from the upper half of each wait. It stops when the job's timeout runs out: the circuit's timeout, else job-timeout-seconds, else 300 seconds. The timeout now counts from the first submission attempt, so a rate-limit wait comes off it rather than adding to it. A cancel ends a wait at once. Any other failure is raised at once, as before.
• Fewer status polls. Unless a circuit names its own poll_interval, the first wait between status polls is the device's new poll-interval-seconds, else 1 second. Each poll that finds the job unfinished doubles it, up to 60 seconds, which is how IQM's own QDMI library polls (IQM_QDMI_device_job_wait). A circuit that names a poll_interval, such as examples/measure_shim_execution.py sends, is polled at that fixed interval, as before. Both drivers poll this way, and no wait runs past the job's deadline. The key is in both whitelists, and a value that is not a number fails before anything is submitted.
• max-concurrent-jobs is a new optional device-access key. It caps how many of a device's circuits are at the provider at once. The shim QPM sets it as the device profile's max_provider_queue_depth, so the controller holds the rest queued and dispatches each as another finishes (SCHED-007). It is in both whitelists, select_qpu and resolve_descriptor. Unset, the device adds no limit of its own. A value that is not a whole number of at least 1 stops the service at start.
• Docs: qfw-device-access.yaml(5), a new "Providers that rate-limit" section in the shim provider guide, the frontend contract's key list, and the benchmarking span table, which names the new rate-limited span event.

Why

On janus1 on 2026-10-23, chemistry_he job 76 failed with QRMI task_start failed: [QrmiError_] error in response: status code 429 Too Many Requests. A rerun that sent one circuit at a time (job 82) got past task_start and then failed with QRMI task_status failed: ... 429 Too Many Requests. So the q20 limits the request rate, and the shim was polling every second. IQM blocks an account that runs over its request quota for 30 seconds. QDMI-on-IQM 1.5.0 documents the block and the 10-second quota window. A status poll can be refused during the block as well as a submission, so the retry covers polls and the result fetch too, and polling backs off to spend less of the quota.

Findings

A shim QPM already sends one job at a time. The controller's operator limit max_inflight defaults to 1 (scheduler_control["dispatch_depth"], since c953bf2), and nothing in QFw or the cluster raises it. So the estimator's three circuits did not reach IQM at once from one QPM. Job 76's 429 more likely came from the account's request quota. Every request counts against it: status polls, introspection through QRMI and QDMI, and any other client on the same token. That makes the retry the fix. max-concurrent-jobs is the per-device half of SCHED-007, and it matters once an operator raises max_inflight. The new end-to-end test pins both behaviours.

There is no Retry-After to honor. QRMI 0.26.0 to 0.26.4 drops the response headers. A 429 arrives as the base qrmi.QrmiError, with the status only in the message:

Provider	How QRMI reports a 429
IQM	error in response: status code 429 Too Many Requests. The generated client keeps no headers, and the message drops the body too.
IBM QRS and QCS	response body: 429 Too Many Requests <body>
Pasqal Cloud	Status: 429 Too Many Requests, Fail <body>
IBM Quantum System	Retried inside QRMI without a limit (iqp_retry_policy), so QFw never sees it and the job timeout does not bound it.

QDMI cannot take a 429 retry. MQT Core reports a failed call as Submitting job: <QDMI status>. or Checking job status: <QDMI status>., without the HTTP status, so the shim cannot tell a 429 from any other failure. QdmiDriver._poll_job still ends a job on its first failed job.check(). On IQM that comes only after QDMI-on-IQM has retried the 429 itself, inside a request timeout of an hour. The QDMI driver does take the new poll schedule. The libraries handle most of the rest themselves:

Library	429 handling
iqm-qdmi 1.5.0 (pinned)	Retries up to 10 times, waiting IQM's Retry-After (30 s if absent), and slows down before the quota runs out.
amazon-braket-qdmi 1.3.1	The AWS SDK's retry strategy, 10 attempts, throttling-aware. It also treats CreateQuantumTask's SERVICE_QUOTA_EXCEEDED as throttling.
ibm-qdmi 1.0.0	Retries a GET twice, honoring Retry-After, but never retries a POST. A rate-limited job submission fails.

Not covered

• target() and the introspection calls still fail on a 429. target() is cached per reservation credential, so that is one call per reservation.
• task_stop stays best effort. A stop during a block can fail, and the job then finishes at the provider.
• Not run against the q20 (the tunnel is down) or on janus1.

Tests

tests/mock/test_shim_rate_limit.py is new, with 40 tests. A fake QRMI resource refuses calls a set number of times, and a fake clock stands in for the waits.

• A 429 in each vendor's wording reads as a rate limit, and other failures do not.
• A submission refused three times goes through on the fourth, after waits of 2, 4 and 8 seconds.
• The retry gives up when the job timeout runs out, with the last wait cut to what is left. A client timeout bounds it too. Without a timeout (Pasqal) it retries until the job goes through.
• A failure that is not a 429 is not retried. A cancel ends a wait with nothing submitted.
• Refused status polls are waited out, and polls refused until the deadline stop the job. Time spent waiting to submit comes off the job timeout. A refused result fetch is waited out.
• Status polls back off 1, 2, 4 ... 60 seconds. The device's poll-interval-seconds is the first wait, a client's poll_interval stays fixed, the last poll is made when the timeout runs out, and a bad value fails before submission. The QDMI driver's _poll_job backs off the same way.
• max-concurrent-jobs becomes the device profile's limit, and a bad value stops the service. An end-to-end run of the real shim QPM with stub drivers shows circuits over the limit queued and dispatched as others finish, under the default max_inflight and a raised one.

tests/mock/test_descriptor_config.py takes max-concurrent-jobs and poll-interval-seconds through select_qpu and resolve_descriptor. tests/mock/test_qdmi_profiles.py checks that QDMI's run_circuit polls on the same schedule.

Run in the test cluster's c1 against this branch:

Stage	This branch	main (27f1b7e)
ci-syntax	passed	passed
mock	881 passed, 1 skipped	835 passed, 1 skipped
unit	132 passed	132 passed
tests/qiskit without test_qdmi_adapters.py	51 passed, 1 skipped	51 passed, 1 skipped

test_qdmi_adapters.py needs the released adapters, which the c1 venv lacks on main too. The new tests passed 15 runs in a row and 8 runs under parallel load.