Repository navigation
perf(agc): pre-sync the DCC key writers of a dispatch's images before the GPU mutex - #6631
Merged
Merged
Conversation
11 tasks done
Contributor
Author
|
Performance report captures for this PR and #6632 (Minecraft Dungeons II title screen, 3840x2160, RTX 5070 Laptop GPU, driver 617.42, Windows 11, warm caches; only
Summary lines ( {"commit":"3a84d7c90ec6","dirty":false,"env":{"APS5_BATCH_WRITEBACK_WINDOWS":"0","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TEXTURE_CACHE_MIB":"3072","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"60:10"},"gpu":"NVIDIA GeForce RTX 5070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"seconds 60:10","first_present":1641,"ms":10009.716},"presents":81,"frame_ms":{"p50":120.725,"p90":138.825,"p99":152.702,"max":152.702},"gpu_busy_ms_per_present":79.229,"worker_busy_pct":{"queue worker 0x0":99.328,"queue worker 0x20":99.104},"top_phases_ms_per_frame":{"Driver.Packet":245.067,"Driver.Packet.dispatch":80.018,"Driver.Dispatch":78.807,"Driver.Packet.memory_wait":56.850,"Vulkan.Dispatch":43.633,"Driver.Packet.draw":42.647,"Driver.Packet.device_lock":42.642,"Driver.Draw":42.514},"trace_dropped":false,"trace":"aps5-trace-3a84d7c90ec6-31640.json.gz"}cl-final: {"commit":"184c4f6ff78a","dirty":false,"env":{"APS5_BATCH_WRITEBACK_WINDOWS":"0","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TEXTURE_CACHE_MIB":"3072","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"60:10"},"gpu":"NVIDIA GeForce RTX 5070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"seconds 60:10","first_present":1607,"ms":10032.248},"presents":80,"frame_ms":{"p50":120.053,"p90":153.035,"p99":176.428,"max":176.428},"gpu_busy_ms_per_present":78.503,"worker_busy_pct":{"queue worker 0x0":99.252,"queue worker 0x20":99.363},"top_phases_ms_per_frame":{"Driver.Packet":248.864,"Driver.Packet.dispatch":90.453,"Driver.Dispatch":89.677,"Driver.Packet.memory_wait":54.234,"Driver.Packet.device_lock":41.955,"Vulkan.Dispatch":40.617,"Vulkan.Dispatch.shader_resources":37.133,"Driver.Packet.draw":35.767},"trace_dropped":false,"trace":"aps5-trace-184c4f6ff78a-32848.json.gz"}cl-main: {"commit":"3a84d7c90ec6","dirty":false,"env":{"APS5_BATCH_WRITEBACK_WINDOWS":"0","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TEXTURE_CACHE_MIB":"3072","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"60:20"},"gpu":"NVIDIA GeForce RTX 5070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"seconds 60:20","first_present":1653,"ms":20084.022},"presents":166,"frame_ms":{"p50":118.371,"p90":133.261,"p99":151.945,"max":154.821},"gpu_busy_ms_per_present":79.556,"worker_busy_pct":{"queue worker 0x0":99.263,"queue worker 0x20":99.341},"top_phases_ms_per_frame":{"Driver.Packet":240.130,"Driver.Packet.dispatch":76.829,"Driver.Dispatch":75.653,"Driver.Packet.memory_wait":57.848,"Driver.Packet.draw":42.803,"Driver.Draw":42.668,"Vulkan.Dispatch":41.119,"Driver.Packet.device_lock":40.975},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":12.057,"events_lost":174434,"totals_full":false}},"trace":"aps5-trace-3a84d7c90ec6-34472.json.gz"}cl-both: {"commit":"97a668ea2080","dirty":false,"env":{"APS5_BATCH_WRITEBACK_WINDOWS":"0","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TEXTURE_CACHE_MIB":"3072","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"60:20"},"gpu":"NVIDIA GeForce RTX 5070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"seconds 60:20","first_present":1675,"ms":20017.651},"presents":180,"frame_ms":{"p50":109.582,"p90":119.944,"p99":130.642,"max":131.130},"gpu_busy_ms_per_present":78.416,"worker_busy_pct":{"queue worker 0x0":99.200,"queue worker 0x20":99.614},"top_phases_ms_per_frame":{"Driver.Packet":220.934,"Driver.Packet.dispatch":95.835,"Driver.Dispatch":95.194,"Driver.Packet.memory_wait":45.714,"Vulkan.Dispatch":41.171,"Vulkan.Dispatch.shader_resources":38.749,"Resources.Complete":37.321,"Driver.Packet.device_lock":30.603},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":13.164,"events_lost":138652,"totals_full":false}},"trace":"aps5-trace-97a668ea2080-32748.json.gz"}Digests: AI-assisted: yes |
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
Contributor
|
A second machine for this PR and #6632: i5-14400F, RTX 4060 8 GB, 16 GB DDR4, Windows 10, Minecraft Dungeons II title screen. Libs
GPU busy stays at 98-100 ms per frame in every arm, so the gain is the idle gap closing; the two levers add up (-10.7 and -4.6 ms alone, -14.2 ms together). 0 fatal and 0 wait timeouts in all 12 runs. AI-assisted: yes (Claude Code) |
Contributor
Author
|
A second machine, measured by @parl0v: i5-14400F, RTX 4060 8 GB, 16 GB DDR4, Windows 10. Minecraft Dungeons II title screen, 200 s rows with
The frame time falls from 151.7 to 137.6 ms with both. Nearly all of that is idle gap, while GPU busy moves by 2 ms. There were no fatal lines and no wait timeouts in the 12 runs. The full table and method are here: AlxcBTW#56 (comment) |
9 tasks done
… the GPU mutex The stage-A pre-sync of a dispatch listed only the surfaces that stage B reads on the CPU. The DCC keys of an image that a recorded kernel writes were not in the list, so stage B waited for that kernel in WaitForKeyWriters with the GPU mutex held and every other worker queued behind the wait. At the Minecraft Dungeons II title the pre-sync fired once in 296,000 dispatch checks while WaitForKeyWriters synchronized 400 times per 10 s. PresyncSurfaces now also returns the key range of every image binding (PresyncSurface::dccKeys). presync waits for a key range only while recorded work writes it, the key-store memo does not already give its value (DccKeyWritersPending) and, under the mutex it takes for a moment anyway, the pending stores do not give the keys either (DccKeyWriterNeedsSync, the test WaitForKeyWriters makes before it synchronizes). Which keys the dispatch reads, and when, is unchanged: the locked lookup runs as before and finds nothing pending. APS5_NO_DCC_PRESYNC=1 leaves the key ranges out. The [presync] line counts key ranges checked, pre-syncs that only the keys required and the outcomes of WaitForKeyWriters. The test agc_driver_dcc_key_presync covers a kernel that writes the keys, a recorded fill of the keys and nothing pending, and skips with the reasons check_skips.py expects when the device has no timeline semaphores or no host import.
AlxcBTW
force-pushed
the
perf/dcc-key-presync
branch
from
October 16, 2026 12:52
a447c96 to
2a15be0
Compare
Contributor
Author
|
Rebased onto Fork CI, Linux with lavapipe, every AI-assisted: yes |
This was referenced Oct 16, 2026
Collaborator
|
I went through this with #6632 and it reads as a safe optimisation: stage B still calls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Companion: #6632 (independent of this one; the two together give more than the sum, see Tested).
A compute dispatch builds its resources in two stages: stage A (binding plan, buffers, descriptor set) runs without the GPU mutex, stage B (the image lookups) under it. Between them
VulkanDevice::presyncwaits, still without the mutex, for recorded GPU work that writes a surface stage B would read on the CPU, so that the locked lookup finds nothing pending and the other workers do not queue behind the wait. That list held only the texel surfaces. The DCC keys of an image are read on the CPU as well (TextureClearKeys, the clear-code decode of a sampled texture or a storage image), and when recorded work writes them with values the driver can't derive from the pending stores,WaitForKeyWriterssynchronizes with the GPU under the mutex.In Minecraft Dungeons II (PPSA16064) on its title screen (3840x2160, about 8 FPS) the pre-sync fired once in 296,000 dispatch checks while
WaitForKeyWriterssynchronized 400 times per 10 s, 1.17 s of waiting under the mutex per 10 s, about 90 % of it on the compute queue worker. The queue 0 worker and the draw committer waited 2.6 s per 10 s behind exactly the compute worker's holds.Graphics::PresyncSurface(address, bytes,dccKeys) replaces the (address, bytes) pair thatShaderResources::PresyncSurfaces,Recipe::presyncSurfacesandVulkanDevice::presyncpassed around.PresyncSurfacesadds the key range (one key byte per 256 surface bytes,DccKeyBytes) of every image binding with a DCC address, for the built object, the cached object and the address-based decode alike.presyncwaits for a key range only while recorded work writes it (the lock-free pending-write snapshot), no key-store memo entry already gives its value (Graphics::DccKeyWritersPending, the same memoPendingStoreKeysconsults) and, under the mutex it takes for a moment anyway, the pending stores do not give the keys either (Graphics::DccKeyWriterNeedsSync: a pending write overlaps andPendingStoreKeyshas no value, the testWaitForKeyWritersmakes before it synchronizes). Which keys a dispatch reads, and when, is unchanged: the locked lookup runs as before and finds nothing pending. A writer recorded after the check still takes the locked wait.[presync]line (APS5_PROFILE_DRAW) counts key ranges checked, pre-syncs that only the keys required, and the outcomes ofWaitForKeyWriters(no pending writer, keys known, fills leaving uncompressed, synchronized).VulkanDevice::PresyncCounts()exposes the first two for the test.APS5_NO_DCC_PRESYNC=1leaves the key ranges out (APS5_NO_PRESYNC=1already disables the whole pre-sync).docs/dev/ARCHITECTURE.md: one paragraph on stage A, the pre-sync and the key ranges.Tested
Windows 11, MinGW-w64 GCC 15.2, NVIDIA GeForce RTX 5070 Laptop GPU (driver 617.42) for the title rows; lavapipe on Linux CI for the tests. The title rows ran on this change on top of
44df21632; the branch is squashed onto5d9dcc42d(the 59 commits in between merge without a conflict and the changed lines are the same).agc_driver_dcc_key_presync(new): a kernel that copies data over the DCC keys of a sampled texture is recorded. Control: a dispatch built under the mutex waits for it inWaitForKeyWriters(syncedrises). WithPrepareDispatch: stage A lists the key range and waits once before the mutex (PresyncCounts:keyChecks,keyWaits,presyncs), the dispatch then makes no locked synchronization, and both samples read the same keys (the texture reads as the 0000 clear the kernel stored). A secondPrepareDispatchwith nothing pending waits for nothing, and one behind a recorded constant fill of the keys, whose value the pending stores give, waits for nothing and the dispatch makes no locked synchronization either. The test skips with the reasonstools/check_skips.pyexpects when the device has no timeline semaphores or no host import. Fork CI (Linux, lavapipe, ubuntu-latest; this squashed head onorigin/main5d9dcc42d): https://github.com/AlxcBTW/AnyPS5/actions/runs/38817872674, 459 of 459 tests of^(agc_driver_|agc_recorder|agc_dcc|agc_display_buffer_dcc|agc_pending|agc_key)pass,agc_driver_dcc_key_presyncin 0.15 s.libsandrelinkercompile on the fork CI (MinGW-w64 GCC): https://github.com/AlxcBTW/AnyPS5/actions/runs/38817909024 (this squashed head).tools/check_conventions.py --base origin/mainpasses.APS5_TEXTURE_CACHE_MIB=3072,APS5_PROFILE_DRAW=1,APS5_PROFILE_GPU=1,APS5_NO_PAD_INPUT=1; themainarm is the binary with both switches off, which an earlier session measured against a build of origin/main: 7.80 against 7.93 FPS):main:APS5_NO_DCC_PRESYNC=1 APS5_HANDOFF_SPIN_US=0APS5_HANDOFF_SPIN_US=0APS5_NO_DCC_PRESYNC=1Rows run in the order both, this PR, companion, main, main, companion, this PR, both. The warm-up row and the first
bothrow, the first two after the relink, found no title windows (probably cold caches), sobothhas one row of eight windows. Rows of identical code differ by up to 0.8 FPS (the twomainrows read 7.08 and 7.62, the two rows of the companion alone 7.16 and 7.97), about 10 %, so a single row cannot resolve an effect of that size; the pooled arms and the repetition in other sessions can. The same scene on the same machine in two earlier sessions: origin/main against the first commit of this PR with the companion, alternating rows, 7.93 against 8.76 FPS (+10.5 %); an earlier lab build, levers off against both levers on, 7.94 against 8.76 (+10.2 %), with this PR's lever alone at +2.6 % and the companion's alone at +2.9 %. The baseline therefore lands between 7.35 and 7.94 FPS from session to session, and the changed build between 8.55 and 8.76 FPS: +10 % to +16 %, +12 % on average of the three sessions. Alone, this PR gives +2.6 % and +4.0 %, the companion +2.9 % and +3.0 %; together they give more than the sum, and the GPU busy time stays at about 80 ms while its idle time per frame falls from 54.7 to 36.3 ms.APS5_TRACE_SECONDS=60:10is set besidesAPS5_TEXTURE_CACHE_MIB=3072,APS5_NO_PAD_INPUT=1and the trace directory; same machine, scene and session, one capture per build, no event dropped): origin/main, 81 presents in 10.0 s = 8.09 FPS, frame time p50 120.7 / p90 138.8 / p99 152.7 ms, GPU busy 79.2 ms; both PRs, 80 presents in 10.0 s = 7.97 FPS, p50 120.1 / p90 153.0 / p99 176.4 ms, GPU busy 78.5 ms. These two captures do not show the difference: a capture is 80 frames of one row, and rows of identical code differ by more than the effect (see above). The captures of the earlier session (20 s,APS5_TRACE_SECONDS=60:20, the queue 0 worker's event buffer full after 12 s, so they are flaggedtrace_dropped) read 8.27 FPS (p99 151.9 ms) for origin/main and 8.99 FPS (p99 130.6 ms) for the first commit of this PR with the companion (+8.7 %). The summary lines of the four captures follow as a comment; the traces are kept and available on request.PendingFillsLeaveUncompressedwould have answered in the locked lookup (it needs the texture's key encoding, which the surface list does not carry; the counter of those answers was 0 in every row), and it skips a range whose key-store memo entry is older than a later unknown writer (the locked lookup then waits as before).Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)