Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,10 @@ smoothly and quickly.
GitHub, which will trigger a GitHub Actions run that you can use to verify
everything is passing.

- For changes that affect **performance**, reviewers may ask for benchmark
results. See [`benchmark/README.md`](benchmark/README.md) for the commonly
used ways to run the benchmarks and compare a change against a baseline.

- Note that there are two GitHub actions checks that need not be green:

1. We test the freshness of the generated proto code we maintain via the
Expand Down
94 changes: 94 additions & 0 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Benchmarks

This directory contains benchmarks for gRPC-Go. They fall into two groups:

1. **`benchmain`** — the primary, configurable benchmark driver. Use this when
you want to measure end-to-end RPC performance and compare a change against a
baseline. This is what reviewers most often ask for on performance PRs.
2. **Go micro-benchmarks** (`go test -bench`) — standard `testing.B` benchmarks
for small, focused pieces of code (e.g. `primitives`, `latency`).

## `benchmain`: the main benchmark driver

`benchmark/benchmain/main.go` runs configurable client/server workloads and
writes a result file you can format and diff.

Run a set of workloads with profiling enabled:
Comment on lines +13 to +16

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since benchmark is a separate Go module (it has its own go.mod), running go run or go test commands targeting the benchmark directory from the repository root will fail with a Go module error:

go: directory benchmark/... is contained in a module that is not the main module

It is highly recommended to instruct users to first change their working directory to benchmark/ before running any commands.

Consider adding a note like this:

First, change your working directory to the `benchmark` directory:

```sh
cd benchmark


```sh
go run benchmark/benchmain/main.go -benchtime=10s -workloads=all \
-compression=gzip -maxConcurrentCalls=1 -trace=off \
-reqSizeBytes=1,1048576 -respSizeBytes=1,1048576 -networkMode=Local \
-cpuProfile=cpuProf -memProfile=memProf -memProfileRate=10000 -resultFile=result
Comment on lines +19 to +22

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Remove the benchmark/ prefix from the path to main.go to align with running the commands from within the benchmark directory.

Suggested change
go run benchmark/benchmain/main.go -benchtime=10s -workloads=all \
-compression=gzip -maxConcurrentCalls=1 -trace=off \
-reqSizeBytes=1,1048576 -respSizeBytes=1,1048576 -networkMode=Local \
-cpuProfile=cpuProf -memProfile=memProf -memProfileRate=10000 -resultFile=result
go run benchmain/main.go -benchtime=10s -workloads=all \
-compression=gzip -maxConcurrentCalls=1 -trace=off \
-reqSizeBytes=1,1048576 -respSizeBytes=1,1048576 -networkMode=Local \
-cpuProfile=cpuProf -memProfile=memProf -memProfileRate=10000 -resultFile=result

```

Pass `-h` to see all available flags:

```sh
go run benchmark/benchmain/main.go -h

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Remove the benchmark/ prefix since the command should be run from within the benchmark directory.

Suggested change
go run benchmark/benchmain/main.go -h
go run benchmain/main.go -h

```

### Comparing against a baseline

The recommended workflow when working on a performance change is to capture a
baseline before your change and compare against it afterward:

```sh
# On the base branch, capture a baseline result file:
go run benchmark/benchmain/main.go -benchtime=10s -workloads=all -resultFile=basePerf

# After your change, capture the current result:
go run benchmark/benchmain/main.go -benchtime=10s -workloads=all -resultFile=curPerf
Comment on lines +38 to +41

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Remove the benchmark/ prefix from the paths to main.go to align with running the commands from within the benchmark directory.

Suggested change
go run benchmark/benchmain/main.go -benchtime=10s -workloads=all -resultFile=basePerf
# After your change, capture the current result:
go run benchmark/benchmain/main.go -benchtime=10s -workloads=all -resultFile=curPerf
go run benchmain/main.go -benchtime=10s -workloads=all -resultFile=basePerf
# After your change, capture the current result:
go run benchmain/main.go -benchtime=10s -workloads=all -resultFile=curPerf

```

Then use `benchresult` to format or compare the result files:

```sh
# Format a single result file:
go run benchmark/benchresult/main.go curPerf

# Compare two result files (prints the change for benchmarks present in both):
go run benchmark/benchresult/main.go basePerf curPerf
Comment on lines +48 to +51

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Remove the benchmark/ prefix from the paths to benchresult/main.go to align with running the commands from within the benchmark directory.

Suggested change
go run benchmark/benchresult/main.go curPerf
# Compare two result files (prints the change for benchmarks present in both):
go run benchmark/benchresult/main.go basePerf curPerf
go run benchresult/main.go curPerf
# Compare two result files (prints the change for benchmarks present in both):
go run benchresult/main.go basePerf curPerf

```

## `run_bench.sh`: shell harness

`benchmark/run_bench.sh` builds the standalone `server` and `client` binaries
and sweeps over combinations of parameters. Each parameter accepts a
comma-separated list of values, and the script runs every combination.

```sh
./benchmark/run_bench.sh -r 1,10 -c 1,10 -req 1,1024 -resp 1,1024 -rpc_type unary,streaming

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Remove the benchmark/ prefix from the path to run_bench.sh to align with running the commands from within the benchmark directory.

Suggested change
./benchmark/run_bench.sh -r 1,10 -c 1,10 -req 1,1024 -resp 1,1024 -rpc_type unary,streaming
./run_bench.sh -r 1,10 -c 1,10 -req 1,1024 -resp 1,1024 -rpc_type unary,streaming

```

Run `./benchmark/run_bench.sh -h` for the full list of options. Defaults are a
warm-up of 10s and a duration of 60s, with single RPC/connection/request/
response sizes and `unary` RPCs.

## Go micro-benchmarks

Some packages under `benchmark/` (for example `primitives` and `latency`)
contain standard Go benchmarks. Run them with the usual `go test` tooling:

```sh
# Run all micro-benchmarks in a package:
go test -bench=. -benchmem ./benchmark/primitives/

# Run a single benchmark:
go test -bench=BenchmarkCodeStringSwitch -benchmem ./benchmark/primitives/
Comment on lines +75 to +78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Remove the benchmark/ prefix from the package paths to align with running the commands from within the benchmark directory.

Suggested change
go test -bench=. -benchmem ./benchmark/primitives/
# Run a single benchmark:
go test -bench=BenchmarkCodeStringSwitch -benchmem ./benchmark/primitives/
go test -bench=. -benchmem ./primitives/
# Run a single benchmark:
go test -bench=BenchmarkCodeStringSwitch -benchmem ./primitives/

```

## Directory overview

| Path | Purpose |
| -------------- | -------------------------------------------------------------- |
| `benchmain/` | Primary configurable benchmark driver (start here). |
| `benchresult/` | Formats a result file and compares two result files. |
| `run_bench.sh` | Shell harness that sweeps parameter combinations. |
| `client/` | Standalone benchmark client binary used by `run_bench.sh`. |
| `server/` | Standalone benchmark server binary used by `run_bench.sh`. |
| `worker/` | Worker driver for the cross-language gRPC OSS benchmarks. |
| `stats/` | Statistics collection and histogram helpers used by the above. |
| `latency/` | Simulated network latency, with its own micro-benchmarks. |
| `primitives/` | Micro-benchmarks for low-level primitives. |
| `flags/` | Shared flag types (e.g. comma-separated int slices). |
Loading