Skip to content

fix(gs): decode sceGsSetDefDBuff trailing args via EABI registers, not o32 stack offsets - #186

Draft
smmathews wants to merge 1 commit into
ran-j:mainfrom
smmathews:feature/23-gssetdefdbuff-eabi-args
Draft

fix(gs): decode sceGsSetDefDBuff trailing args via EABI registers, not o32 stack offsets#186
smmathews wants to merge 1 commit into
ran-j:mainfrom
smmathews:feature/23-gssetdefdbuff-eabi-args

Conversation

@smmathews

@smmathews smmathews commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Problem

sceGsSetDefDBuff reads its ztest, zpsm and clear arguments from stack offsets 16, 20
and 24. Under this runtime's EABI those three are register-passed in $t0$t2
(GPRs 8–10), so the reads pick up caller-frame bytes and seed the double buffer's Z-test
and Z-format from stale stack data on every call.

Fix

Swap the three readStackU32 calls for decodeGsTrailingArgs3, which reads the registers
first and falls back to the old stack offsets only when all three registers are zero.

That helper already exists and is already used by sceGsSetDefDBuffDc for the identical
three arguments — this makes the two siblings consistent. Because the fallback triggers
only on an all-zero register triple, the sole behavioural change is for register-passed
nonzero trailing arguments, which is the buggy case.

Basis

The EE calling convention passes eight integer arguments in GPRs 4–11. The test for any
readStackU32 site is how many register arguments precede it: fewer than eight is a bug,
eight or more is correct.

sceGifPkRefLoadImage in the same file also calls readStackU32, at offsets 0 and 8, and
is left untouched — it consumes eight register arguments first (GPRs 4–11), so its width
and height genuinely are the ninth and tenth arguments and belong on the stack.

#195 fixes the same defect class in sceSifSendCmd.

Testing

One added test, sceGsSetDefDBuff decodes trailing args from the recompiler register ABI:
it seeds distinct sentinels in GPRs 8–10 and at the o32 stack offsets, then asserts the
written ZBUF and TEST fields carry the register values rather than the stack bytes.

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_FLAGS=-msse4.1 -DCMAKE_CXX_FLAGS="-msse4.1 -include cstdint"
cmake --build build
./build/ps2xTest/ps2x_tests

The -include cstdint is needed to build this branch as it stands: the vendored ELFIO
header omits it and newer libstdc++ no longer supplies the typedefs transitively. #199
fixed that on main, so the flag becomes unnecessary once this branch is rebased.

Mutation: revert the three-line helper swap and the two new assertions report the stack
sentinels instead of the register sentinels.

Risk and not in scope

  • No repo-wide audit of other readStackU32 sites was done. SIF.cpp and
    MemoryCard.cpp also call it; auditing them is separate follow-up work.
  • The all-zero-register fallback preserves the previous behaviour exactly for callers that
    genuinely pass zeros in all three registers, so such a caller sees no change.

…t o32 stack offsets

sceGsSetDefDBuff read ztest/zpsm/clear with readStackU32 at offsets
16/20/24. Under the EABI this runtime targets, arguments 5-8 arrive in
$t0-$t3 (GPRs 8-11), so those stack offsets hold caller-frame bytes, not
the trailing args. The sibling sceGsSetDefDBuffDc already decodes the
same three args correctly with decodeGsTrailingArgs3 (registers first,
stack only as an all-zero fallback); this routes sceGsSetDefDBuff through
the same helper. Adds a register-ABI regression test that seeds distinct
register and stack sentinels and asserts the written ZBUF/TEST fields
reflect the register values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant