agc-driver: storage image write-back drops GPU results for CPU-stamped 64 KiB edge blocks (boxes in Silksong's menu text)

Summary

StorageTexture write-back keeps the CPU's bytes for a whole 64 KiB block when a CPU stamp it cannot place falls in it. In PPSA12544 (Hollow Knight: Silksong, Unity TextCore) this discards GPU-generated glyphs, so some menu letters render as solid boxes. On the title menu, the "O" of OPTIONS is one; the options menu has many. APS5_EAGER_WRITEBACK=1 (or APS5_NO_TEXTURE_CACHE=1) avoids it.

What happens

Silksong builds its SDF font atlas on the GPU:

A 512x512 R8 storage image (0x40000 bytes, not 64 KiB aligned; for example 0x25e042000) is filled with Unity's allocator pattern 0xCD.
A 64x64x1 dispatch writes the whole atlas, followed by one small dispatch per glyph (2x4, 4x5, ...). Each reads a CPU-rasterized glyph bitmap (another 512x512 R8 texture) and writes the glyph's SDF into the atlas. APS5_TRACE_DISPATCH_IO:
```
[dispatch-io] 64x64x1: texture 0x25f9fe000+0x40000(512x512 f1 t2) ... storage 0x25e042000+0x40000(512x512 f1 t2) ...
[dispatch-io] 1x4x1:   texture 0x25f9fe000+0x40000(512x512 f1 t2) ... storage 0x25e042000+0x40000(512x512 f1 t2) ...
```
Before those results are stored, the tracker holds newer CPU stamps for the block the atlas starts in. In writeBackLayers the first unit is an edge block (atlas bytes +0x0..+0xe000; the atlas starts 0x2000 into its 64 KiB block). StoredOver returns true for it, so the block is kept for the CPU. The [gpu] storage image ...: guest memory changed while GPU results were pending; keeping the CPU's blocks warning appears for this image.
The kept block re-uploads from guest memory, which is still 0xCD there. The SDF of a glyph in that area becomes near-solid, and that letter renders as a box.

Two findings from instrumenting it:

Most of the edge stamps come from other allocations. StoredOver returns true as soon as cpuStampOf(block) > generation, but the CPU stamp covers the whole 64 KiB block. A store to another allocation sharing the block counts as a store over the surface. The atlas's tail edge block (+0x3e000..+0x40000) was kept only for that reason. I recorded the generation per 4 KiB page from the GetWriteWatch pages / Linux runs and only treated pages overlapping [from, to) as stored over. That kept the tail block's results.
The remaining stamps change nothing. The head block still had newer CPU stamps on the atlas's own first two pages (page stamps 87615 6945 96596 96596 87617 ... 87617, layer generation 87617). Dumping those 8 KiB at the decision shows they are all still 0xCD. So whatever the CPU did there left the bytes equal to the fill, or happened before the GPU writes and was collected later. Storing the GPU results over that block (a local experiment) renders every glyph correctly.

So, at least for this title, block-granular "the CPU wins" loses GPU results when the CPU stamp is not ordered after the GPU work. I have not found the writer of those two pages, and I am not sure what rule you would want here. Some options: track pages instead of blocks; collect CPU writes of a storage image's range when a dispatch marks it pending, so earlier stores get older stamps; or compare against a baseline. Happy to test any variant against this title.

Repro

PPSA12544 on Windows 11 with an RX 9070 XT; main plus the bring-up stack in Hollow Knight: Silksong (PPSA12544) #726.
Boot to the title menu. The "O" of OPTIONS and a mark after "GAME" are boxes.
With APS5_EAGER_WRITEBACK=1 they render correctly.
APS5_NO_SAMPLED_FROM_STORAGE, APS5_NO_DRAW_RESOURCE_CACHE, APS5_NO_RESOURCE_CACHE, APS5_NO_DISPATCH_CACHE, APS5_NO_COLLECT_MEMO, APS5_NO_FILL_HLE, APS5_NO_FILL_CLEAR, APS5_CPU_COPIES and APS5_SYNC_TEXTURE_UPLOAD each leave the boxes.

(Report written with AI assistance.)

Follow-up: APS5_EAGER_WRITEBACK=1 is not a clean workaround. It fixes the boxed letters, but the digit glyphs (the version string under the Team Cherry logo, the volume values in the Audio menu) then render with a thin rectangle around each glyph cell. They are clean on a default run and with APS5_NO_TEXTURE_CACHE=1. Locally I instead stored the GPU results for CPU-stamped edge blocks in writeBackLayers (skipping the StoredOver check for edges). With that, both the letters and the digits render correctly. I am not proposing that as the fix, since the edge rule exists for a reason, but it narrows the problem to the edge-block keep decision.

More impact from the same rule, in gameplay. After walking from the first room of PPSA12544 into the next section and back, a large dark foreground layer (a black silhouette with a soft green glow) stays fixed on screen while the world scrolls behind it. During the session the log repeats storage image ...: guest memory changed while GPU results were pending; keeping the CPU's blocks (7 times). With APS5_EAGER_WRITEBACK=1 the silhouette still appears on entering but fades out after a few seconds, as it presumably should, so the layer seems to be a GPU-written transition or darkness effect whose later updates are dropped by the keep-CPU-blocks decision. APS5_NO_RESIDENT_TARGETS, APS5_NO_SAMPLED_FROM_STORAGE and APS5_NO_DRAW_RESOURCE_CACHE each leave it stuck.

A correction to my earlier comment: storing GPU results over CPU-stamped edge blocks is not just unprincipled. Two runs with that local change crashed in Unity's TLSF allocator (free-list unlink on a corrupted block). I have since seen the same crash once without it, so the evidence is mixed, but I would not use that change.

My current guess at the root cause is ordering. In the glyph-atlas case the CPU stamps on the atlas's own pages carried bytes still equal to the allocator fill (0xCD). That suggests the writes happened before the GPU work, and the pages were only collected (stamped with a newer generation) after the results became pending. Collecting a storage image's pages when its layers become pending (before the dispatch is recorded) would give such stores an older stamp than the pending results. I have not tried that yet.

Picked up the finding 1 half here, because it falls out of the code's own driver path rather than needing a rule decision.

StoredOver already compares exact range pieces for driver stores; the CPU branch read one generation per 64 KiB block, so a store to another allocation sharing the block counted for the surface. #1387 makes the CPU branch range-precise the same way (each collect records its dirty pages; only the pages the range overlaps are tested), and flips the shared-block assertion in agc_write_tracking, which fails on main:

without:  write tracking test failed: a CPU store outside the shared range counts for it
with:     write tracking tests passed

Full suite 408/408. It does not address finding 2 — the stamps on the surface's own pages whose bytes are still 0xCD — so if that is what keeps the head block, the boxes can survive this. I have no PPSA12544 dump, so I ran no title; the fix is offered for your repro to say whether the tail-block half holds and whether the letters change.

On the rule you asked about: of your three options, this is the narrowest — it keeps the block-granular rule everywhere except where the block is only partly the surface, where the exact answer already existed for driver stores. If you would rather have "collect when a dispatch marks pending" or a baseline comparison instead, say so and I will not press this one.

Written with AI assistance.

@Dextroid17

Thanks @Dextroid17. I tested #1387 (e920c9e, cherry-picked onto my integration build) with PPSA12544 on Windows 11, without any write-back workaround:

Menu text: the letters in the options menus are still boxed. The digits are clean, as on a default run.
Dark layer after room changes: after walking into the next section and back, it still appears and stays on screen. It does not fade, which is worse than with eager write-back of the 1920x1080 images.
Frame rate settles at about 60 fps.

So the tail-block half alone doesn't change what's visible here. That points at finding 2, the stamps on the surface's own pages. Your fix still looks right to me for the shared-block case, and I'd keep it.

Next, I'll try collecting a storage image's pages when its layers become pending, before the dispatch is recorded. Then CPU writes that happened before the GPU work get an older stamp than the pending results. I'll report back here with the result.

More data, with #1387 applied and APS5_TRACE_FLUSH=1, no workaround. In the run, the menu showed boxed letters, then a room change left the dark layer stuck.

Glyph atlas (0x25dfd2000+0x40000, 512x512 R8, tile 2; neither end is 64 KiB aligned):

[gpu] storage image 0x25dfd2000: guest memory changed while GPU results were pending; keeping the CPU's blocks
[flush] image 0x25dfd2000+0x40000 keeps 2 CPU-written 64 KiB blocks (first stored byte at +0xe000, 1 ranges stored, generation 93472)

The two kept blocks are the head and tail edge blocks. The head block holds the atlas's first 0xe000 bytes (its top rows), where the boxed glyphs are. The boxes are flat light grey, which matches the 0xCD allocator fill I saw in those pages before. So even with page-precise stamps, a CPU stamp on the atlas's own pages wins, and it looks like the allocation fill rather than glyph data.

I also tried collecting the upload and refresh generations uncached (a local switch; the memo there can be older than the uploaded bytes). It made no visible difference, so stale memoized generations at upload are not the cause.

Full-screen layer: not an edge case at all:

[flush] image 0x2631d0000+0x900000 for sampled texture 0x26385e000+0x10000
[flush] image 0x2631d0000+0x900000 keeps 144 CPU-written 64 KiB blocks (first stored byte at +0x900000, 0 ranges stored, generation 94060)
[flush] image 0x263b10000+0x900000 for sampled texture 0x263881000+0x800000
[flush] image 0x263b10000+0x900000 keeps 88 CPU-written 64 KiB blocks (first stored byte at +0x580000, 1 ranges stored, generation 94303)
[gpu] storage image 0x263b10000+0x900000: memory no longer registered; 1 pending ranges dropped
[flush] image 0x266750000+0x870000 keeps 9 CPU-written 64 KiB blocks (first stored byte at +0x0, 2 ranges stored, generation 652922)
[gpu] storage image 0x266750000+0x870000: memory no longer registered; 2 pending ranges dropped

Here whole 64 KiB blocks of 9 MiB images are kept for the CPU, all 144 in one case. Each flush is triggered by sampling a different, smaller texture inside the image's memory, so the title reuses that memory for other resources. Two images were also unregistered while results were pending. So the stuck layer is probably about memory reuse, or results dropped on unregister, not the edge rule. I haven't yet identified which of these images is the layer itself. I'll correct my earlier comment that tied the dark layer to the edge-block decision.

Full log (114 lines) available on request.

fix(agc): decide an edge unit's CPU keep from the pages the unit over…

Found the cause of the stuck dark layer. It is not the CPU-blocks keep rule. This corrects my last comment and the original title: the boxed glyphs remain a separate problem.

Mechanism. The layer is drawn into a 1920x1080 format-56 render target with DCC (tile 7, resident image). Every frame the title fast-clears it by writing a clear code into the key range, then draws into it. A resident target's Refresh sees a fast clear only as a change of the key value in guest memory. Draws leave their results pending on the GPU and do not touch the keys, so after a draw the keys in memory still hold the clear code. The next fast clear writes the same code, Refresh sees no change, and the clear is never applied: the old contents stay in the image. On hardware the draw replaces the clear codes of the blocks it writes, so the next clear is a real change. Eager write-back hides this, because each write-back marks the keys uncompressed.

Bisection (PPSA12544, each run one room change):

eager write-back only for 1920x1080 format 56: the layer fades;
the same, limited to the draw path (MarkDirty on a draw's residents): fades. Limited to compute writes: stuck. No render-target MarkDirty call from the target-binding path fired for these images;
no eager write-back, the fast-clear drop rule off, the unregistered drop off, or uncached generation collects: stuck;
every sampled read of these targets is a live view of the pending image, not a snapshot. Copies and buffer reads never touched them (flush trace).

Check. No eager write-back. After MarkDirty on a draw's resident image, a local switch marks its DCC keys uncompressed (MarkDccUncompressed) when its uploaded keys are a clear code. The layer fades as on hardware, with no other visible change, at 60 fps (eager write-back costs about a third of that).

Writing the keys on every draw is a CPU store per draw, so I'd rather not propose it as is. The narrower fix is in the refresh: a write to the key range since the image last saw its keys, while results are pending, is a new fast clear even when the value is unchanged. The key-range stamps already exist (KeyProof). I'll open a PR with a test that fast-clears, draws, fast-clears with the same code and samples the target, unless someone familiar with the key handling prefers a different route.

Update: no PR needed for the dark layer. #903 (merged) stores uncompressed keys at the first write of a fast-cleared image, which is the same fix in the right place. My integration build predated it. With #903 cherry-picked and no other switch, the layer no longer sticks in PPSA12544, at 60 fps. So the dark layer is fixed on main.

The boxed glyphs are still there with #903, and they are a separate problem: the head and tail edge blocks of the unaligned 512x512 R8 glyph atlas keep CPU bytes that look like the allocator fill (see my comment above). This issue now only covers that part.

@Dextroid17

The glyph boxes have a cause too: the CPU "writes" that win the atlas's edge blocks don't change the bytes.

Timeline for the 512x512 R8 atlas (tile 2, 0x25e142000+0x40000, neither end 64 KiB aligned), from a lifecycle trace at startup:

upload reason=first;
172 compute writes rasterize glyphs into it (pending);
at 2.36 s, Refresh sees CPU stamps on its pages (cpuWrote 1, originalValid 0);
the write-back keeps both edge blocks for the CPU (StoredOver true on the atlas's own pages, fix(agc): decide an edge unit's CPU keep from the pages the unit overlaps #1387 applied) and stores the middle;
upload reason=cpu reloads the image from guest memory. The glyphs in the first 0xe000 bytes are gone, and the grey boxes are the bytes that were there before.

Experiment. A local switch keeps a copy of the bytes each upload started from (and of the result after each write-back) for images up to 1 MiB. In blocks kept for the CPU, it stores a GPU byte wherever the GPU changed it and the CPU did not (guest byte still equal to the copy); the CPU wins where both changed a byte. For the atlas it reports 65512 GPU bytes merged into 2 CPU-kept ranges, so nearly every byte the GPU drew there was untouched by the CPU. With it, the menu text renders correctly with no eager write-back, at 60 fps. (It forces the CPU write-back path for those cases, since the direct path never sees the results on the CPU.)

So the keep decision trusts the write watch, which reports page writes, not changes. equalsOriginal exists for this, but original is not held after a GPU-direct upload, which is why originalValid is 0 here.

A possible fix, narrower than my switch: for an edge unit only (at most two partial blocks per layer), keep the uploaded bytes of the edge pieces. When an edge block would be kept for the CPU, merge byte-wise against them as above, instead of dropping the GPU's bytes. Full blocks keep the current rule. The cost is up to 2 x 64 KiB per image with unaligned ends, plus a CPU readback of those pieces only when a merge is needed.

@Dextroid17, does that fit how you see the edge rule, or would you rather avoid keeping bytes per image? I can open a PR with a test (GPU writes into an unaligned surface, then a CPU store of identical bytes on its edge page, then a sample), unless someone prefers a different direction.

Opened #1599 for finding 2: the CPU-keep decision now compares the stamped range with the guest bytes the image had at its generation (images up to 1 MiB), and stores the GPU results where they are unchanged. It is the range-level form of the baseline idea above, without forcing the CPU write-back path.

Checked against PPSA12544 on Windows 11 (RTX 5090) with #846, #1482, #1495 and #788 for bring-up: the title menu, Options and Game Options have no boxed glyphs, and APS5_TRACE_FLUSH=1 reports no CPU-kept blocks. A regression test in agc_driver_recorder_tests reproduces the head edge (0xe000 bytes dropped without the fix).

(Written with AI assistance.)

@merwie

@merwie, I opened #1918 following your suggestion to keep baselines for just the partial edge blocks and merge byte by byte. It preserves real CPU edits alongside the GPU results and keeps the full-block rule.

The edge regression and Vulkan synchronization validation pass on Linux, but I haven’t tested PPSA12544. If you have time, a check against your menu-text repro would be helpful. The PR also notes the overlap with #1599.

Partly addressed by #1599 (and #903 for the dark layer): the CPU-keep decision compares the stamped range with the guest bytes the image had at its generation and stores the GPU results where the CPU left them unchanged, covered by unchangedCpuStampTests in agc_driver_recorder_tests (without the fix dropped 14336 of 65536 texels the GPU wrote). Still open: partial edge blocks where the CPU really edits bytes next to GPU results (#1918, closed unmerged).

Confirming the remaining open case on a second title, very different from Silksong's: RIDGE RACER (PPSA31372), a PS1 classic running under the libpsemu emulator, Windows 11 (10.0.26200), RTX 3050 6GB (NVIDIA 617.42), main eaae7e069.

During a race the road texture is rewritten every frame; the dropped GPU results show as a blue flicker on the road. Static scenes (menus) are unaffected. Bring-up report: #8196

One extra data point on the workaround discussion: APS5_EAGER_WRITEBACK=1 is not usable on this title — race loading collapses to sub-5 FPS because the write-back path runs every frame. So the eager path trades the flicker for a frame-rate cliff on dynamic-VRAM titles, and the byte-by-byte edge merge (#1918's direction) is what would fix this class properly.

Navigation Menu

Repository navigation

agc-driver: storage image write-back drops GPU results for CPU-stamped 64 KiB edge blocks (boxes in Silksong's menu text) #865

Summary

What happens

Repro

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

agc-driver: storage image write-back drops GPU results for CPU-stamped 64 KiB edge blocks (boxes in Silksong's menu text) #865

Description

Summary

What happens

Repro

Activity

merwie commented on Oct 6, 2026

merwie commented on Oct 6, 2026

Dextroid17 commented on Oct 8, 2026

merwie commented on Oct 8, 2026

merwie commented on Oct 8, 2026

merwie commented on Oct 8, 2026

merwie commented on Oct 8, 2026

merwie commented on Oct 8, 2026

andrey231 commented on Oct 8, 2026

nikhilpanchal1 commented on Oct 9, 2026

adriadam10 commented on Oct 10, 2026

authurtabo-byte commented on Oct 20, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions