Repository navigation
fix(shader): write cache files of 2 GiB and more in chunks - #7127
Merged
oneandonlydean merged 1 commit intoOct 17, 2026
Merged
Conversation
On Windows (MinGW libstdc++) one std::ofstream::write of 2 GiB or more never returns: the whole temporary file is written but WriteFileAtomically hangs, the entry is never replaced and the temporary file stays behind. A Vulkan pipeline cache of 2.1 GB hung every save and left a 2 GB .tmp per run. Write in chunks of at most 1 GiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBXM4pgkMHYEtqN1EQni2P
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
11 tasks done
Contributor
|
The new Written with AI assistance (Claude Code). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On Windows (MinGW libstdc++),
WriteFileAtomically(core/shader/recompiler/ShaderCacheDirectory.hpp) never returns when it writes 2 GiB or more. It does onestd::ofstream::writeofbytes.size(). The whole temporary file reaches the disk (2 GiB + 4096 bytes in the test), but the call never comes back. The temporary file is then never renamed over the entry and is never removed.Where it happens: in GTA V (PPSA04264) the Vulkan pipeline cache file (
vk-pipelines-*.bin) grew to 2.1 GB. After that:.tmp.*file next to the cache (36 GB of them before we noticed);Fix: write in chunks of at most 1 GiB and stop after the first chunk that fails. The existing
file.close()/file.fail()check still decides the result. On Windows, reads already go inMAXDWORDchunks (ReadWholeFile, af2b869), so this PR only changes the write side.Related: #6771 (keeps the pipeline cache file with its shader cache version) slows how fast that file grows, but any cache file that reaches 2 GiB within one version still hangs its save without this change. #6958 edits
PipelineCache.cpp; this PR does not.Test:
agc_shader_disk_cache_large_file(agc_shader_disk_cache_tests --large-file) writes 2 GiB + 4096 bytes throughWriteFileAtomicallyon a worker thread and waits up to 120 s. It then checks the file size, six sampled bytes on both sides of the 1 GiB and 2 GiB boundaries, and that no temporary file is left, then deletes the file. If the write does not return in 120 s, the test fails at once instead of hanging ctest (ctestTIMEOUT 240as a backstop). It returns 77 (skip) when it cannot allocate the buffer or when there is less than 3 GiB of free disk space in the temp directory.Tested
Windows 11 Pro, WinLibs GCC 15.2.0 (posix-seh, ucrt), Release, i9-13900K / 64 GB, rebased on
mainc9e6c45:main(fix reverted):agc_shader_disk_cache_large_filefails withWriteFileAtomically did not return within 120 s for a file of more than 2 GiB, with a full-sizepipelines.bin.tmp.*left on disk;agc_shader_disk_cachetest passes too;.tmpfiles left, and Story loads in ~30 s with no pipeline compile stalls.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)🤖 Generated with Claude Code
https://claude.ai/code/session_01XBXM4pgkMHYEtqN1EQni2P