Repository navigation
feat(libc): scan 16-bit wide conversions in fscanf on Linux - #15
Closed
KernelPanicNT wants to merge 2 commits into
Closed
KernelPanicNT wants to merge 2 commits into
KernelPanicNT wants to merge 2 commits into
Conversation
KernelPanicNT
force-pushed
the
feat/libc-fscanf-guest-wide-conversions
branch
2 times, most recently
from
October 14, 2026 18:47
5c33c30 to
cdf9322
Compare
On Linux, sscanf, vsscanf, sscanf_s and vsscanf_s threw for %ls, %lc, %l[, %S and %C because glibc stores a 32-bit wchar_t per character into the guest's 16-bit buffer. A format with one of these conversions is now scanned directive by directive in ScanGuest: each wide directive reads its bytes as the narrow directive does and widens every byte through this library's mbrtowc, which is what FreeBSD 11.4's vfscanf.c reads in the single-byte C locale of none.c. The byte buffer for a wide directive is bounded by the remaining input, not only by the guest's width. As in FreeBSD, %S and %C are wide whatever length modifier precedes them. The format pre-scan and ScanGuest recognise the q length modifier as FreeBSD does, so %lS, %hC and %qS are found too instead of reaching glibc, and %qd or %qn in a format scanned by ScanGuest stores a long long. On Windows, ScanGuest hands q to the host as ll for integer conversions. A narrow s, c or [ is handed to the host without any length modifier other than l, so %lls, %qs or %Lc in a format scanned by ScanGuest are narrow, as in FreeBSD, instead of reaching glibc as wide. A wide conversion combined with a positional argument or the ' flag still throws. fscanf keeps throwing for wide conversions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fscanf threw for %ls, %lc, %l[, %S and %C on Linux. The body of ScanGuest moves into ScanGuestSource, which reads from a source: a string, as before through ScanGuest, or a stream, where each conversion other than %n is one glibc fscanf call and the scanner peeks with getc/ungetc under the stream's lock. fscanf uses the stream source for formats with a wide conversion, so a wide directive reads its bytes as the narrow one and widens them through this library's mbrtowc, as sscanf already does. On the stream, wide directives take their bytes with the POSIX.1-2008 m assignment-allocation modifier, since a stream's string length is not known before it is read. The string source keeps its buffer bounded by the remaining input. Every byte the stream conversion stored is widened, so a NUL byte read by %ls or %l[ is kept as a 0 unit, as FreeBSD's convert_wstring and convert_wccl store it. m needs glibc, so on other non-Windows hosts fscanf throws std::runtime_error for a wide conversion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KernelPanicNT
force-pushed
the
feat/libc-fscanf-guest-wide-conversions
branch
from
October 15, 2026 13:16
cdf9322 to
e32dea4
Compare
Owner
Author
|
Superseded: published upstream; closing the CI draft. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Depends on #14, which adds 16-bit wide scanning to
sscanf,vsscanf,sscanf_sandvsscanf_son Linux. This PR is the single commit on top of it. On Linux,fscanfstill throws for%ls,%lc,%l[,%Sand%C(RejectGuestWideScan, called fromfscanf), because glibc would store a 32-bitwchar_tper character into the guest's 16-bit buffer. This PR scans them from the stream.Source of the behaviour. No console measurement is involved. The model is the same as in the PR this depends on, now read from a stream:
release/11.4.0, commit8c63994)vfscanf.cL161-L308 and L683-L728: withl,%c,%[and%sread the stream one character at a time through the locale'smbrtowc.%S/%Care%s/%cwithl(L594-L612). FreeBSD'svsscanfruns the same__svfscanfon a stringFILE(vsscanf.cL61-L71), sofscanfandsscanfshare these semantics.none.cL99-L113: the C-localembrtowcmaps one byte to its value, as this library'smbrtowcdoes.massignment-allocation character of POSIX.1-2008fscanf(added in Issue 7, SD5-XSH-ERN-132; accepted by glibc's__isoc99_vfscanf). The number of bytes a stream directive reads is not known in advance, andmmakes glibc allocate exactly that many. This is host plumbing only; the guest sees no allocation.Changes:
ScanGuestbecomesScanGuestSource, a template over an input source:StringScanSourceholds the existing string cursor. Thesscanffamily on both platforms (including the Windows_sforms) goes through it with unchanged behaviour.StreamScanSource(Linux only) reads a hostFILE*. Literals and whitespace are matched with onegetc/ungetcpair per peeked byte. Each conversion is one glibcfscanfcall with a trailing%n.%ncounts the bytes the call consumed.fscanfsends a format with a 16-bit wide conversion toScanGuestSourceover the stream. Formats without one still go to glibc'svfscanfunchanged, andSyncStatusstill runs after either path.%<width>m<narrow conversion>. glibc allocates the bytes, which are widened throughmbrtowcas in the string case and then freed.%lsand%l[are terminated,%lcis not.RejectGuestWideScanhad no other caller and is removed.Still throwing, and why:
%1$ls) or the'flag throwsstd::runtime_errorbefore reading the stream, as in the PR this depends on.fscanftranslates the format for the host, whosewchar_tis 16 bits.%Sand%Cstill throw in that translation.Tests:
guest_standard_streamsgainsCheckWideScanon Linux, run ontmpfilestreams. Each case also checks the next byte left in the stream throughfgetc:%ls%n %3lc%donab cdef 12: 2 assignments,%nis 2,%dfails onf, andfis left.%C+%lcona-b: 1 assignment, and-is left.%l[a-z]on12: returns 0, the buffer is untouched, and1is left.%Son bytes\xe9t\xff: U+00E9,t, U+00FF.EOF, with the guest EOF flag set.%*ls%donab: returns 0.%1$ls: throws with the buffer and the stream untouched.Progress (
tools/progress.py, base8ffbb49plus the PR this depends on): system libraries 3244/3671 → 3245/3671 (+fscanf_nid_postfix). The tool countedfscanfas unimplemented because its single definition reachedNotImplemented_nid_no_patchon Linux.Tested
Head
c4a894ac4cf5df8aa164b647e1cae8442196cc35, base3a94783e170a1797f6074c5fa9128c77050f437f.BUILD_TESTING=ON, tests run under QEMU 8.2 user mode with-cpu EPYC-Romebecause the build host lacks AVX2):guest_standard_stream_tests5/5,guest_formatting_tests5/5 passes.core/libs/prx/libc/src/Formatting.cppwith the base version:guest_standard_stream_testsfails withfscanf 16-bit wide conversion on Linux not implemented;guest_formatting_testsstill pass there, as expected for regression coverage. Restoring the PR source and rebuilding passesguest_standard_stream_tests5/5,guest_formatting_tests5/5.git diff --checkpass. The Linux job of the Build workflow now targets the project's self-hosted runner, so it was not run on the fork; no title was run.Duplicate and overlap check
fscanfconversions on Linux by decoding UTF-8. It stacked on feat(libc): scan 16-bit wide conversions on Linux boykopovar/AnyPS5#5285 and was closed by a maintainer with "no title is known to reach this". This PR names no title either (see Unblocks). fix(libc): throw for 16-bit wide scanf conversions on Linux boykopovar/AnyPS5#3984 introduced the throw this PR replaces.fscanf, its NIDnpLpPTaSuHgand "wide scanf". No open PR implements wide scanning infscanf. feat(libc): implement wide and unlocked stream I/O boykopovar/AnyPS5#5816 adds wide stream input (fgetwc,fgetws,getwc,getwchar) inFormattingWide.cpp/StandardStreams.cpp, not scanf conversions.git merge-file) of each PR head against this branch, per file, from the PR's merge base.fscanf_nid_postfixsits above the changed lines.Formatting.cpp.core/libs/tests/GuestStandardStreams.cpp.core/libs/tests/Formatting.cpp.docs/dev/debt/libc.md.int main() {right afterCheckDescriptorAccessModesincore/libs/tests/GuestStandardStreams.cpp, where this PR insertsCheckWideScanand its call.core/libs/tests/Formatting.cpp) and feat(libc): implement collate<wchar_t>::_Getcat boykopovar/AnyPS5#5662 (thecollate<char>line next to the scanf line indocs/dev/debt/libc.md).mainindocs/dev/debt/libc.md.main. The branch merges into84b01f9without conflicts.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)