Skip to content

Use SOMAXCONN for the server TCP listener backlog - #229

Open
derekste wants to merge 1 commit into
epics-base:masterfrom
derekste:dev/listener-backlog
Open

derekste wants to merge 1 commit into
epics-base:masterfrom
derekste:dev/listener-backlog

Conversation

@derekste

@derekste derekste commented Oct 1, 2026 •

Copy link
Copy Markdown

When 16 independent PVA client contexts connect simultaneously, the server's four-slot TCP accept queue can overflow and delay some clients beyond the burst fixture's five-second initial-monitor deadline. Use SOMAXCONN when creating the listener so the platform can provide its supported queue depth.

@shreya-fermi reproduced this on Linux arm64 and proposed the change in fermi-ad/redis-pvxs-ioc#122. A separate Linux amd64 comparison on adlinux3 confirms the result:

Listener backlog Fresh 16-context bursts Listener Send-Q Namespace ListenOverflows
4 5 passed, 15 timed out; 10–15 of 16 initially ready on failures 4 74 total; 0–6 per cycle
SOMAXCONN 20 passed, 0 failed 4096 0 on every cycle

The variants were interleaved for 20 cycles each, with fresh private Redis, IOC, and client containers on an internal bridge with no host ports. Both runtime images use the same IOC binary and dependencies; the rebuilt PVXS library differs by this listener constant. IOC UID 10001, read-only root, 256 MiB/128 PID limit; client 512 MiB/128 PID limit. No OOM kills occurred.

The unchanged burst client at 0eb4e76 uses one UInt32 scalar PV, 16 independent contexts, pipelined monitors with queue size 16, an explicit UDP address, and automatic address discovery disabled. After hurryUp(), it publishes a zero sample and waits five seconds for all initial monitors, then sends one validation sample. This fixture deadline is not a product startup guarantee.

A/B source: IOC 47626c7d8a31, PVXS 8e00eaecdee5. The same four-slot constant remains on current upstream a9f8b7b, which this PR targets. Runtime image IDs: backlog 4 sha256:94b8146263f38c75e9527f619d6ed5ae0ef3edd1165d88bb3c6e2fc64dfaa167; SOMAXCONN sha256:ee66ac7f001c204d536c624c38099ed8baf63933fa0a2c3116c3e28c60b56d69.

Validation: both A/B builders pass all 19 IOC CTest cases. The patch on current upstream passes all 25 standalone PVXS core suites (2490 assertions) on Linux amd64. On macOS arm64, 24 core suites pass; UDP forwarding fails identically with the original and patched listener. EPICS database IOC suites were outside these core runs. The platform constant compiles on both Linux and macOS.

The source-host evidence archive is retained at /mnt/newdrive/derekste/pvxs-backlog-review-20260930-evidence.tar.gz, SHA256 57faf3d039721dc6bc658bb3da75a6871e3e3f9792c9baf70c9cbdf8d170f3fb. The downstream issue stays open through upstream review and final candidate burst/reconnect qualification.

Upstream GitHub CI: at this update, all five workflow runs are marked action_required with no jobs created. Native validation is complete; hosted CI needs action by the upstream maintainers.

@mdavidsaver

Copy link
Copy Markdown
Member

@derekste Thank you for keeping this to a one line change. Please try to be similarly concise with the PR description.

When 16 independent PVA client contexts connect simultaneously, the server's four-slot TCP accept queue can overflow and delay some clients beyond the burst fixture's five-second initial-monitor deadline.

This is the desired behavior when a server is being subjected to a DoS attack. Perhaps not by intent, but in effect. I am open to changing the magic 4 to something else, but I do not want to disable this limit completely. Certainly not on RTEMS where file descriptors are precious.

Looking on my Debian 13 host.

$ grep -wR SOMAXCONN /usr/include/
...
/usr/include/bits/socket.h:#define SOMAXCONN    4096
/usr/include/x86_64-linux-gnu/bits/socket.h:#define SOMAXCONN   4096

imo. 4096 is too large.

$ grep -wR SOMAXCONN /usr/x86_64-w64-mingw32/include/
...
/usr/x86_64-w64-mingw32/include/winsock2.h:#define SOMAXCONN 0x7fffffff

And 0x7fffffff is waaaaay too large.

$ find /opt/rtems/5 -name include | xargs grep -wR SOMAXCONN
/opt/rtems/5/i386-rtems5/include/sys/socket.h:#define   SOMAXCONN       128

Even 128 seems too large on RTEMS.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants