feat(videoout): use native Wayland on Linux

[alfalfa-star]

What

On Linux the window now runs natively under a Wayland compositor instead of through XWayland.

• CMakeLists.txt: build SDL's Wayland backend (SDL_WAYLAND ON) next to X11. SDL's defaults are kept: libwayland-client/-cursor/-egl, xkbcommon and libdecor are loaded with dlopen at run time, so the binary gains no new link-time dependencies.
• VideoOutDriver: on Linux, set SDL_HINT_VIDEODRIVER to wayland,x11 before SDL_InitSubSystem(VIDEO). SDL2 lists X11 first in its bootstrap order, so without the hint it picks XWayland even in a Wayland session.
• Fallbacks: SDL skips the Wayland driver when there is no WAYLAND_DISPLAY (and the session isn't wayland) or the libraries are missing, then uses X11. An SDL_VIDEODRIVER environment variable still wins over the hint, so SDL_VIDEODRIVER=x11 gets XWayland back. The ctest flip tests, which set SDL_VIDEODRIVER=offscreen, also keep their driver.
• CI (build.yml, release.yml): install libwayland-dev libxkbcommon-dev libegl-dev libdecor-0-dev, so the Linux release includes the backend. Without these packages configuration still succeeds and the build has X11 only.
• docs/dev/BUILD.md: lists the packages (Debian/Ubuntu and Fedora) and the SDL_VIDEODRIVER=x11 override.

Overlap: #3574 edits the same apt-get lines in both workflows and the same BUILD.md line. Whichever lands second only needs both package lists joined.

Tested

Arch Linux, KDE Plasma (Wayland), GCC, Release, RADV (RX 9070 XT), on main a11c8f7.

• LEGO Horizon Adventures (PPSA14632): the process maps libwayland-client, libdecor and xkbcommon and not libX11. The swapchain is created on the Wayland surface (1536x864, 3 images), and the game renders until main's known stop (color views of several array slices are unsupported, Layered rendering: a draw into all slices of a 3D color target aborts with "color views of several array slices are unsupported" #2102), the same as under X11.
• SDL_VIDEODRIVER=x11, and a run with WAYLAND_DISPLAY unset and XDG_SESSION_TYPE=x11: libX11 is loaded, libdecor is not, and the swapchain is created as before.
• video_out_flip_tests (lifetime, reopen, controls, overlay) with ANYPS5_REQUIRE_DISPLAY=1 and no SDL_VIDEODRIVER, so they run on native Wayland: all pass.
• ctest: 832/832 (the 5 display/device tests skip as on main). tools/check_conventions.py passes.

🤖 Generated with Claude Code

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes

Assisted by Claude Code

[Dextroid17]

All three claims that decide whether this is safe check out against the vendored SDL2.

• No new link-time dependencies. 3rdparty/SDL2/CMakeLists.txt:475-478 has SDL_WAYLAND defaulting on for Unix and SDL_WAYLAND_SHARED defaulting to ON — "Dynamically load Wayland support" — so the backend is resolved at runtime and a build without the packages stays X11-only.
• A comma list is the right syntax. SDL_video.c:522, inside SDL_VideoInit, reads SDL_GetHint(SDL_HINT_VIDEODRIVER) and walks it with while (driver_attempt && *driver_attempt != 0 && !video), splitting on , at :523 and trying each bootstrap in turn — so "wayland,x11" is a bootstrap order rather than a single name, which is what makes the X11 fallback inside the same string work instead of needing separate logic. (The similarly-shaped walk at :4517 belongs to SDL_ShowSimpleMessageBox, not to video init.)
• The hint is set in the right place, and that is not accidental here. It goes into the constructor immediately before SDL_InitSubSystem(SDL_INIT_VIDEO | SDL_INIT_GAMECONTROLLER), beside the existing SDL_HINT_WINDOWS_DPI_AWARENESS write. The constructor is reachable only through the function-local static VideoOutDriver instance in Get(), so C++11's thread-safe one-time initialisation makes it a single, race-free write before SDL starts at all. A video-driver hint set anywhere later would be too late, and this placement avoids that by construction.
• Using SDL_SetHint rather than SDL_SetHintWithPriority is also what keeps the environment override working, since a hint at normal priority loses to the same name in the environment.

One thing I would ask about, because it changes behaviour for every Linux user rather than only Wayland sessions. Preferring Wayland when the session provides it is clearly right. The question is whether the default should flip in this PR or behind a documented option, given that the fallback paths are only exercised by some sessions — no WAYLAND_DISPLAY, no libdecor, or a compositor that constrains window size differently. The description already documents SDL_VIDEODRIVER=x11 as the way back, which is what makes either choice defensible; I would just want the flip to be a deliberate decision rather than a side effect of adding the backend.

The note about #3574 touching the same apt-get lines and the same BUILD.md line is the right thing to have written down, and joining both package lists is the obvious resolution.

Written with AI assistance.

[nicolasfalesy]

Tested on Hyprland (Omarchy, Arch) with an Intel GPU (Core Ultra X7 358H, Xe, Mesa 26.2.2 ANV), Release, this PR (b02ef22) merged onto main 37dd483 (only the workflow and BUILD.md conflicts, resolved to main for a local build). SDL_WAYLAND configures ON and SDL_VIDEO_DRIVER_WAYLAND is defined.

Native Wayland, no X server and no DISPLAY in the session, catalog releases relinked with --to-intel, 25 s each:

Title	Result
ScreenTester5 (PPSA69420)	native Wayland window (app.elf), overlay drawn
DOOM Native Port (PPSA99666)	native window, title menu drawn, 59.94 FPS
OpenRCT2 (PPSA99728)	native window, its game-files screen drawn, 59.94 FPS
Prospero Vibrate (PPSA99920)	native window, 19.99 FPS

All four were still running at 25 s; screenshots match what they draw under XWayland.

One caveat about the documented override: SDL_VIDEODRIVER is also visible to titles that ship their own SDL. With SDL_VIDEODRIVER=x11 (and a working X server on DISPLAY), OpenRCT2 aborts at start-up with import: no video: x11 not available from its bundled SDL, while DOOM runs normally. I hit the same on main with the variable set for the host. So the BUILD.md line may want a note that the override can break titles with their own SDL, or the host choice could use a variable the guest doesn't read.

Tested with Claude Code assistance.

[adriadam10]

The change looks right and the third-party runs on KDE and Hyprland cover it. It now conflicts with main:

• .github/workflows/build.yml and .github/workflows/release.yml: the apt-get line now also installs libpipewire-0.3-dev libpulse-dev libasound2-dev; keep those and append libwayland-dev libxkbcommon-dev libegl-dev libdecor-0-dev.
• docs/dev/BUILD.md: the Linux requirements line gained the audio packages; join it with the Wayland text.

Please rebase on current main and push; it can be merged once CI is green.

[alfalfa-star]

Rebased onto current main (0214069) as one commit, 040d2f5.

• build.yml / release.yml: kept main's packages (libpipewire-0.3-dev libpulse-dev libasound2-dev libxrandr-dev), added libwayland-dev libxkbcommon-dev libegl-dev libdecor-0-dev, and kept main's mesa cache steps as they are.
• BUILD.md: kept main's XRandR line and joined the Wayland text into the Linux requirements line beside the audio packages.
• @nicolasfalesy, thanks for the Hyprland/Intel runs. I added your caveat to that line: titles that bundle their own SDL read SDL_VIDEODRIVER too, and some of them fail to start with x11.

Built locally on Linux (Release), and SDL configures with both backends: SDL_VIDEO_DRIVER_WAYLAND (dynamic, with libdecor) and SDL_VIDEO_DRIVER_X11. tools/check_conventions.py passes.

AI-assisted: yes (Claude Code)

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#3891	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4156	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4396	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4397	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4422	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4423	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4424	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4520	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4521	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4522	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4523	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4566	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4599	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4718	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4740	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4749	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4760	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#4813	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#5021	core/libs/prx/libSceVideoOut/src/VideoOutDriver.cpp L263
#5396	.github/workflows/release.yml L65

[alfalfa-star]

Rebased onto current main (e2cd213) as 040873f. #5580 added a Windows-only SDL_HINT_DIRECTINPUT_ENABLED hint at the same spot in VideoOutDriver.cpp. Both are kept: the _WIN32 block and then the __linux__ Wayland hint. Nothing else changed. Built locally (Release, SDL_WAYLAND ON); the video_out tests pass (13/13) and tools/check_conventions.py passes.

Met: rebased onto main at 040873f, keeping #5580's Windows hint before the Linux one.