fix(audioout): reject volumes above 0 dB

[SzymonLeja]

What

sceAudioOutSetVolume now rejects invalid volume values below 0 or above the maximum 32768, returning 0x80260009 (SCE_AUDIO_OUT_ERROR_INVALID_VOLUME) before modifying channel state.

Added a regression test covering negative and oversized volumes and verifying that a previously valid volume remains effective. The behavior is independent of PR #1518, and no other open PR implements the same function change.

Tested

Windows 11. No title or homebrew tested.

• cmake --build build --target audio_out_mix_level_pad_spk_tests --parallel 2 — passed
• ctest --test-dir build -R '^audio_out_mix_level_pad_spk$' --output-on-failure — passed, 1/1
• cmake --build build --target libs --parallel 2 — passed
• python tools/check_conventions.py --base origin/main — passed
• python tools/check_test_registration.py build — passed; no newly unregistered tests
• git diff --check — passed
• git show --check --oneline HEAD — passed

The valid range was cross-checked against compatible Vita SDK and shadPS4/OpenOrbis implementations.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes

[github-actions]

Thank you for your interest in the project! We are sorry, but we cannot accept more PRs while we work through the existing backlog. This PR has been automatically closed and labeled "intake paused". This is not a judgment of the quality of your changes.