Avoid setting recbuf/sndbuf in replica(_reader) processes

[the-mikedavis]

osiris_replica and osiris_replica_reader set recbuf/sndbuf via setsockopt, about 2 MiB for recbuf and 1.4 MiB for sndbuf.

osiris/src/osiris_replica_reader.erl

Line 14 in 492bee9

-define(DEF_SND_BUF, 146988 * 10).

osiris/src/osiris_replica.erl

Line 92 in 492bee9

-define(DEF_REC_BUF, 408300 * 5).

This requests these buffer sizes from the kernel, but the kernel's settings net.core.rmem_max/wmem_max default to about 224 kiB. These kernel settings clamp the requested buffer sizes to those values. Explicitly requesting these options actually does more harm than good. Requesting them at all sets SOCK_RCVBUF_LOCK/SOCK_SNDBUF_LOCK which turn kernel autotuning off for that socket, even though tcp_rmem and tcp_wmem allow growth to several MiB. If the net.core.rmem_max/wmem_max are not increased from their defaults, the replication sockets are clamped down to 224 kiB instead of increased. If we do not set the options, the buffer sizes tune automatically and increase past the 224 kiB default.

This change avoids setting recbuf/sndbuf unless the options are explicitly configured (replica_recbuf/replica_sndbuf app env). This change doesn't affect the setting of the userspace buffers within Erlang.

Testing

On an m7g.16xlarge 3-node cluster, the stock sysctls are:

net.core.rmem_max = 229376        net.core.wmem_max = 229376
net.ipv4.tcp_rmem  = 4096 87380 6291456
net.ipv4.tcp_wmem  = 4096 20480 4194304
net.ipv4.tcp_moderate_rcvbuf = 1

With a single in-VPC producer (c6gn.xlarge) in the same AZ, 8192 byte messages, stock --confirms (10000), a fresh stream per run, four runs per configuration, 60 seconds each, TLS connections terminated by the broker:

scenario	env override	mean MiB/s	Δ vs fix
this change	none	561.1	N/A
pre-change recbuf behavior	replica_recbuf=2041500	285.6	-49.1%
pre-change sndbuf behavior	replica_sndbuf=1469880	508.2	-9.4%

ss on the leader's replication sockets, mid-run:

scenario	cwnd	wscale	rwnd_limited	sndbuf_limited
fix	70–108	7,7	0%	27–33%
replica_recbuf forced	41–42	2,7	98–100%
replica_sndbuf forced	72–75	7,7	0%	26–33%

Separately, if we use Osiris unpatched and tune net.core.rmem_max and net.core.wmem_max together, we see strong growth from small increases, converging on the throughput ceiling at around 1 MiB. This test used three c6gn.xlarge clients all within the same AZ as the writer, all targeting the same stream:

rmem and wmem	mean MiB/s	CV
default (229376)	378.1	1.1%
512 KiB	567.5	0.6%
1 MiB	579.0	0.1%
2 MiB	579.8	0.3%
4 MiB	580.7	0.1%
8 MiB	581.3	0.1%
16 MiB	580.4	0.1%
32 MiB	581.1	0.1%
64 MiB	580.1	0.4%

Publisher confirmation latency also falls by about half with this change at these throughputs.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[the-mikedavis]

I ran iperf3 between these m7g.16xlarge instances and it seems that this 580 MiB/sec is the best we can do on a single connection that spans AZs. Max throughput according to iperf3 on even the network-optimized client instances (c6gn.xlarge) is identical.

Ah, the ~5 Gbps is documented:

When instances are not in the same cluster placement group, bandwidth for single-flow traffic is limited to 5 Gbps.

Let me see how far this can push with ENA Express (25 Gbps).

[the-mikedavis]

Looks like the cap for a single stream is 985-995 MiB/s with ENA express. I got that driven by two clients (same c6gn.xlarge). Adding additional clients doesn't move the needle.