Fix future-wait callback accumulation on every spin path - #1702
Open
hugogo1998 wants to merge 1 commit into
Open
Fix future-wait callback accumulation on every spin path#1702hugogo1998 wants to merge 1 commit into
hugogo1998 wants to merge 1 commit into
Conversation
Every call to future.add_done_callback() appends unconditionally, and every wait path here (spin_until_future_complete's internal loop and both spin_once_until_future_complete implementations) called it fresh on each spin. A single wait that takes N spins to complete registers N callbacks; when the future completes, each becomes its own Task, O(N) memory and work that grows without bound the longer a caller waits. Register one stable wake-on-done callback per Executor instance and add it to a future at most once (checking future._callbacks first), via a new _add_wake_on_done() helper used by all three call sites: spin_until_future_complete, and both SingleThreadedExecutor's and MultiThreadedExecutor's spin_once_until_future_complete. This covers every way a caller waits on a future, including looping spin_once_until_future_complete directly, not just the internal loop inside spin_until_future_complete. Signed-off-by: agentperfopt <295481948+agentperfopt@users.noreply.github.com>
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
skyegalaxy
self-requested a review
July 30, 2026 16:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
future.add_done_callback()appends unconditionally, and every wait path here was calling it fresh on every spin. A singlespin_until_future_complete()that takes N spins to finish was registering N callbacks on the future, and when it completes,Future._schedule_or_invoke_done_callbacks()creates one Task per callback. So waiting N times costs O(N) memory and O(N) work at completion, for no reason. Measured on real Humble rclpy at N=5000: 43.8 ms to complete a future carrying 5000 accumulated callbacks, each one becoming its own Task.#1374 already fixed this for
spin_until_future_complete's internal loop, by rerouting it to a private method that doesn't re-add the callback. That's a real fix and it works: 0.31 ms at N=5000 on that path, matching what this change also gets. Butspin_once_until_future_completeis public API, and a caller who writes their own wait loop around it directly (while not future.done(): executor.spin_once_until_future_complete(future), a completely reasonable and documented way to use it) still hits the original bug, because that method still adds a fresh lambda on every call. Checked this on real Humble rclpy: looping it directly at N=5000 costs 44.7 ms and 5000 callbacks, basically unchanged from before #1374, because the fix never reached this path.This registers one stable wake-on-done callback per Executor instance and adds it to a future at most once, checking
future._callbacksbefore appending, through a single_add_wake_on_done()helper used everywhere a future is waited on:spin_until_future_complete, and bothSingleThreadedExecutor's andMultiThreadedExecutor'sspin_once_until_future_complete. Every spin path gets the same O(1) behavior instead of just one of them.Measured on real Humble rclpy, N=5000:
spin_until_future_completespin_once_until_future_completeloopTies the existing fix on the path it already covers, and closes the gap on the one it didn't, 40x-149x faster depending on N, since the leak scales with how many times a caller has to spin.
The wake-on-done semantics are unchanged: the callback's whole job is to break the executor out of its wait, and firing it once is exactly as sufficient as firing it N times, the future still completes with the correct result either way. This isn't a new mechanism, just making the existing one idempotent everywhere it's used, so it doesn't matter whether a caller uses
spin_until_future_completeor writes their own loop aroundspin_once_until_future_complete, either way they get the fix.Found via an automated perf-optimization pass profiling the future-completion path; the accumulation showed up directly in the profile as O(N) Task creation.