feat(pullsync): delivery batching performance analysis

[janos]

Pullsync Batching Performance Analysis

This PR gives the overview of the performance testing and profiling conducted to evaluate the proposed batched chunk delivery (DeliveryBatch) against the legacy single-message delivery (v2.8.1) in the pullsync protocol. It should not be merged, as it is meant to reproduce the results, but it may be used for potential improvements.

The goal is to identify if the batching of delivery messages in the pullsync protocol can result to a speedup in overall deliveries.

To simulate real-world networking overhead accurately, a benchmark was written that spins up real libp2p nodes locally, multiplexes streams, and transmits chunks between a client and server.

This approach gives a convenient testing, benchmarking and profiling, but the pullsync performance can be measured on the real network as well.

There are two benchmarks:

• BenchmarkDeliveryLibP2P - that uses the complete libp2p stack with real networking
• BenchmarkDeliveryPipe - that just pipes the messages with buffers (no actual networking) which is not the best for real world network conditions but is good for algorithmic changes.

Benchmark

There are to benchmarks, one for he currently released bee v2.8.1 pullsync implementation and one for the proposed batched implementation. Both of them with 25 and 120 chunks delivered. Since the max batch size can be 25 chunks because of the limitation of 128KB payload size of protobuf. There is a possiblity to raise that limit, but with a danger to expose some vulnerability in the network.

Run the libp2p benchmark for 100 iterations tracking memory allocations:

go test -v -run=NONE -bench=BenchmarkDeliveryLibP2P -benchtime=100x -benchmem ./pkg/pullsync/delivery_test.go ./pkg/pullsync/pullsync_test.go

goos: darwin
goarch: arm64
cpu: Apple M1 Pro
BenchmarkDeliveryLibP2P
BenchmarkDeliveryLibP2P/n=25
BenchmarkDeliveryLibP2P/n=25/Batched
BenchmarkDeliveryLibP2P/n=25/Batched-10 	     100	  99077116 ns/op	  601238 B/op	    5915 allocs/op
BenchmarkDeliveryLibP2P/n=25/v2.8.1
BenchmarkDeliveryLibP2P/n=25/v2.8.1-10  	     100	  99083815 ns/op	  507315 B/op	    6076 allocs/op
BenchmarkDeliveryLibP2P/n=120
BenchmarkDeliveryLibP2P/n=120/Batched
BenchmarkDeliveryLibP2P/n=120/Batched-10         	     100	 475344458 ns/op	 2241125 B/op	   27103 allocs/op
BenchmarkDeliveryLibP2P/n=120/v2.8.1
BenchmarkDeliveryLibP2P/n=120/v2.8.1-10          	     100	 475295048 ns/op	 1690773 B/op	   27764 allocs/op
PASS
ok  	command-line-arguments	115.854s

Profiling

Ran Go's built-in pprof tools against both the Batched and v2.8.1 branches of the benchmark independently, to identify any bottlenecks and/or improvements.

Generate CPU and Memory profiles for the Batched implementation:

go test -run=NONE -bench='BenchmarkDeliveryLibP2P/n=120/Batched$' -benchtime=50x -cpuprofile=cpu_batched.prof -memprofile=mem_batched.prof ./pkg/pullsync/delivery_test.go ./pkg/pullsync/pullsync_test.go

Generate CPU and Memory profiles for the legacy v2.8.1 implementation:

go test -run=NONE -bench='BenchmarkDeliveryLibP2P/n=120/v2.8.1$' -benchtime=50x -cpuprofile=cpu_v281.prof -memprofile=mem_v281.prof ./pkg/pullsync/delivery_test.go ./pkg/pullsync/pullsync_test.go

Analyze:

go tool pprof -top cpu_batched.prof | head -n 15
go tool pprof -top cpu_v281.prof | head -n 15
go tool pprof -top mem_batched.prof | head -n 15
go tool pprof -top mem_v281.prof | head -n 15

CPU Profile

Both profiles reveal an identical bottleneck. In both implementations, runtime.pthread_cond_wait, runtime.usleep, and runtime.pthread_cond_signal consume the majority of all execution samples. This indicates the CPU is blocking on libp2p network I/O synchronization than performing protobuf serialization or algorithmic work.

Batched:

      flat  flat%   sum%        cum   cum%
     0.58s 42.96% 42.96%      0.58s 42.96%  runtime.pthread_cond_wait
     0.24s 17.78% 60.74%      0.24s 17.78%  runtime.usleep
     0.22s 16.30% 77.04%      0.22s 16.30%  runtime.pthread_cond_signal
     0.08s  5.93% 82.96%      0.08s  5.93%  syscall.rawsyscalln
     0.05s  3.70% 86.67%      0.05s  3.70%  golang.org/x/crypto/sha3.keccakF1600
     0.05s  3.70% 90.37%      0.05s  3.70%  runtime.kevent

v2.8.1 (Single Message):

      flat  flat%   sum%        cum   cum%
     0.54s 35.06% 35.06%      0.54s 35.06%  runtime.pthread_cond_wait
     0.25s 16.23% 51.30%      0.25s 16.23%  runtime.pthread_cond_signal
     0.20s 12.99% 64.29%      0.20s 12.99%  runtime.usleep
     0.10s  6.49% 70.78%      0.10s  6.49%  syscall.rawsyscalln
     0.09s  5.84% 76.62%      0.09s  5.84%  runtime.memclrNoHeapPointers
     0.08s  5.19% 81.82%      0.08s  5.19%  runtime.kevent
     0.06s  3.90% 85.71%      0.06s  3.90%  golang.org/x/crypto/sha3.keccakF1600

Memory Profile

Batched:

      flat  flat%   sum%        cum   cum%
   27.63MB 22.44% 22.44%    27.63MB 22.44%  github.com/ethersphere/bee/v2/pkg/pullsync/pb.(*Delivery).Unmarshal
   26.50MB 21.53% 43.97%    26.50MB 21.53%  golang.org/x/crypto/sha3.(*state).Sum
   17.63MB 14.32% 58.29%    20.23MB 16.44%  github.com/gogo/protobuf/io.(*varintWriter).WriteMsg
       5MB  4.06% 62.35%        5MB  4.06%  golang.org/x/crypto/sha3.NewLegacyKeccak256 (inline)
    4.42MB  3.59% 65.94%    32.55MB 26.44%  github.com/gogo/protobuf/io.(*varintReader).ReadMsg
    4.23MB  3.43% 69.38%     4.73MB  3.84%  github.com/libp2p/go-buffer-pool.(*BufferPool).Get

v2.8.1 (Single Message):

      flat  flat%   sum%        cum   cum%
   36.16MB 30.47% 30.47%    36.16MB 30.47%  github.com/ethersphere/bee/v2/pkg/pullsync/pb.(*Delivery).Unmarshal
   28.50MB 24.01% 54.48%    28.50MB 24.01%  golang.org/x/crypto/sha3.(*state).Sum
       7MB  5.90% 60.38%        7MB  5.90%  github.com/ethersphere/bee/v2/pkg/bmt.(*goroutineHasher).Write
    4.50MB  3.79% 64.17%    20.01MB 16.85%  github.com/ethersphere/bee-old/pkg/pullsync.(*Syncer).Sync
    4.50MB  3.79% 67.96%     4.50MB  3.79%  golang.org/x/crypto/sha3.NewLegacyKeccak256 (inline)
       3MB  2.53% 70.49%        3MB  2.53%  github.com/ethersphere/bee/v2/pkg/p2p/protobuf.Reader.ReadMsgWithContext

Both the batched and unbatched implementations perform nearly identically over libp2p (~99.07 ms vs ~99.08 ms for n=25 and ~475.34 ms vs ~475.29 ms for n=120).

The separated CPU profiles reveal why this is the case: the top CPU consumers in both implementations are OS-level thread blocking operations (runtime.pthread_cond_wait, runtime.usleep, runtime.kevent). These account for over 60-80% of all CPU samples depending on the run. Because the program is almost entirely bottlenecked by waiting for libp2p stream data to traverse network sockets, any optimizations gained by batching protobuf structures in memory are negligible compared to the network latency.

The memory profile shows that chunk parsing and cryptographic hashing (golang.org/x/crypto/sha3.(*state).Sum and pb.(*Delivery).Unmarshal) dominate memory allocations.

Even if the delimitedReaderMaxSize is raised to 1MB, allowing larger batches, no performance gains are visible.

Conclusion

Introducing a breaking protocol change to batch protobuf messages yields no statistical CPU performance gain over the multiplexer latency, so the simpler, single-message streaming approach is preferred.

I encourage everyone interested to validate the findings.

Checklist

• I have read the coding guide.
• My change requires a documentation update, and I have done it.
• I have added tests to cover my changes.
• I have filled out the description and linked the related issues.

Description

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

AI Disclosure

• This PR contains code that has been generated by an LLM.
• I have reviewed the AI generated code thoroughly.
• I possess the technical expertise to responsibly review the code generated in this PR.

[janos]

One comment regarding TCP (and UDP) Dynamics, a standard internet packet (MTU) is about 1500 bytes. A single 4 KB chunk already requires 3 TCP packets. Whether you flush chunks to the OS individually or in a batch, the OS TCP stack will still stream them out in 1500-byte packets at the maximum rate the network allows.

[janos]

Until the real cluster measurements are performed, I have run a benchmarks on local machine, but using dnctl to simulate the latency on the network interface.

The benchmark that is executed is:

go test -bench=BenchmarkDeliveryLibP2P/n=120 -benchmem -benchtime=5x ./pkg/pullsync

Results without added latency:

goos: darwin
goarch: arm64
pkg: github.com/ethersphere/bee/v2/pkg/pullsync
cpu: Apple M1 Pro
BenchmarkDeliveryLibP2P/n=120/Batched-10               5         387043375 ns/op         2456576 B/op      27140 allocs/op
BenchmarkDeliveryLibP2P/n=120/v2.8.1-10                5         388048475 ns/op         1734056 B/op      27794 allocs/op
PASS

Than a latency is set with:

sudo dnctl pipe 1 config delay 200ms
echo "dummynet out from 127.0.0.1 to 127.0.0.1 pipe 1" | sudo pfctl -f - -e

And the benchmark is run again, the results are:

goos: darwin
goarch: arm64
pkg: github.com/ethersphere/bee/v2/pkg/pullsync
cpu: Apple M1 Pro
BenchmarkDeliveryLibP2P/n=120/Batched-10               5        1622241875 ns/op         2409558 B/op      27160 allocs/op
BenchmarkDeliveryLibP2P/n=120/v2.8.1-10                5        1625406400 ns/op         1730278 B/op      27831 allocs/op
PASS

If you want to reproduce the benchmarks on the local machine, do not forget to remove the latency:

sudo pfctl -F all -d
sudo dnctl -q flush

The results show that the latency has an impact on the performance of the pullsync algorithm, between two runs (with and without latency), but there are no differences between the batched and unbatched version of the algorithm. he latency penalty in pullsync is a TCP/IP layer phenomenon governed by the initial handshake and bandwidth delay, not an application-level delivery issue.

Both implementations suffer from the exact same latency penalty (TCP Slow Start): When a 200ms delay is injected, both versions see their delivery time increase from ~0.38s to ~1.62s. This ~1.2-second penalty is caused by the initial protocol handshake (the Get -> Offer -> Want) and the TCP congestion window (slow start) ramping up across a high-latency link.

Batching application-level data does not improve stream performance: With 200ms of latency, the Batched implementation (1.622s) and the v2.8.1 implementation (1.625s) are essentially identical. This proves that because legacy pullsync already streams data into the socket buffer rather than waiting for individual application-level ACKs, application-level batching provides no benefit. In both cases, the OS segments the stream into identical 1500-byte MTU packets and transmits them at the exact same rate.

While providing no speed benefit, the Batched implementation consistently consumes more memory (2.4 MB/op) compared to the legacy v2.8.1 implementation (1.7 MB/op). This is because the batched version must allocate and hold large arrays of chunk data in memory simultaneously before writing them to the socket buffer.

In this test, the batching protocol change introduces memory overhead without delivering any network speedup compared to the bee v2.8.1 pullsync implementation.