benchmark: add README documenting how to run benchmarks (#8574)#9144
benchmark: add README documenting how to run benchmarks (#8574)#9144luongs3 wants to merge 1 commit into
Conversation
Adds benchmark/README.md covering the common ways to run the benchmarks: the benchmain driver and baseline comparison via benchresult, the run_bench.sh parameter sweep, and the go test -bench micro-benchmarks, plus a directory overview. Links to it from CONTRIBUTING.md so authors and reviewers can find it when a performance PR needs benchmark results. Fixes grpc#8574. RELEASE NOTES: n/a
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9144 +/- ##
==========================================
- Coverage 83.16% 83.09% -0.07%
==========================================
Files 417 417
Lines 33690 33690
==========================================
- Hits 28019 27996 -23
- Misses 4253 4268 +15
- Partials 1418 1426 +8 🚀 New features to boost your workflow:
|
|
@luongs3: Could you please sign the CLA for us to start reviewing the changes. |
easwars
added
Status: Requires Reporter Clarification
Area: Benchmarking
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds documentation for running benchmarks in gRPC-Go, updating CONTRIBUTING.md to reference the benchmark guidelines and creating a new benchmark/README.md file that explains how to use the benchmain driver, run_bench.sh harness, and Go micro-benchmarks. The review feedback correctly points out that because the benchmark directory is a separate Go module, running the documented commands from the repository root will result in a Go module error. The reviewer suggests adding instructions to change the working directory to benchmark/ first and updating all example commands to remove the benchmark/ prefix accordingly.
| `benchmark/benchmain/main.go` runs configurable client/server workloads and | ||
| writes a result file you can format and diff. | ||
|
|
||
| Run a set of workloads with profiling enabled: |
There was a problem hiding this comment.
Since benchmark is a separate Go module (it has its own go.mod), running go run or go test commands targeting the benchmark directory from the repository root will fail with a Go module error:
go: directory benchmark/... is contained in a module that is not the main module
It is highly recommended to instruct users to first change their working directory to benchmark/ before running any commands.
Consider adding a note like this:
First, change your working directory to the `benchmark` directory:
```sh
cd benchmark| go run benchmark/benchmain/main.go -benchtime=10s -workloads=all \ | ||
| -compression=gzip -maxConcurrentCalls=1 -trace=off \ | ||
| -reqSizeBytes=1,1048576 -respSizeBytes=1,1048576 -networkMode=Local \ | ||
| -cpuProfile=cpuProf -memProfile=memProf -memProfileRate=10000 -resultFile=result |
There was a problem hiding this comment.
Remove the benchmark/ prefix from the path to main.go to align with running the commands from within the benchmark directory.
| go run benchmark/benchmain/main.go -benchtime=10s -workloads=all \ | |
| -compression=gzip -maxConcurrentCalls=1 -trace=off \ | |
| -reqSizeBytes=1,1048576 -respSizeBytes=1,1048576 -networkMode=Local \ | |
| -cpuProfile=cpuProf -memProfile=memProf -memProfileRate=10000 -resultFile=result | |
| go run benchmain/main.go -benchtime=10s -workloads=all \ | |
| -compression=gzip -maxConcurrentCalls=1 -trace=off \ | |
| -reqSizeBytes=1,1048576 -respSizeBytes=1,1048576 -networkMode=Local \ | |
| -cpuProfile=cpuProf -memProfile=memProf -memProfileRate=10000 -resultFile=result |
| Pass `-h` to see all available flags: | ||
|
|
||
| ```sh | ||
| go run benchmark/benchmain/main.go -h |
There was a problem hiding this comment.
Remove the benchmark/ prefix since the command should be run from within the benchmark directory.
| go run benchmark/benchmain/main.go -h | |
| go run benchmain/main.go -h |
| go run benchmark/benchmain/main.go -benchtime=10s -workloads=all -resultFile=basePerf | ||
|
|
||
| # After your change, capture the current result: | ||
| go run benchmark/benchmain/main.go -benchtime=10s -workloads=all -resultFile=curPerf |
There was a problem hiding this comment.
Remove the benchmark/ prefix from the paths to main.go to align with running the commands from within the benchmark directory.
| go run benchmark/benchmain/main.go -benchtime=10s -workloads=all -resultFile=basePerf | |
| # After your change, capture the current result: | |
| go run benchmark/benchmain/main.go -benchtime=10s -workloads=all -resultFile=curPerf | |
| go run benchmain/main.go -benchtime=10s -workloads=all -resultFile=basePerf | |
| # After your change, capture the current result: | |
| go run benchmain/main.go -benchtime=10s -workloads=all -resultFile=curPerf |
| go run benchmark/benchresult/main.go curPerf | ||
|
|
||
| # Compare two result files (prints the change for benchmarks present in both): | ||
| go run benchmark/benchresult/main.go basePerf curPerf |
There was a problem hiding this comment.
Remove the benchmark/ prefix from the paths to benchresult/main.go to align with running the commands from within the benchmark directory.
| go run benchmark/benchresult/main.go curPerf | |
| # Compare two result files (prints the change for benchmarks present in both): | |
| go run benchmark/benchresult/main.go basePerf curPerf | |
| go run benchresult/main.go curPerf | |
| # Compare two result files (prints the change for benchmarks present in both): | |
| go run benchresult/main.go basePerf curPerf |
| comma-separated list of values, and the script runs every combination. | ||
|
|
||
| ```sh | ||
| ./benchmark/run_bench.sh -r 1,10 -c 1,10 -req 1,1024 -resp 1,1024 -rpc_type unary,streaming |
There was a problem hiding this comment.
Remove the benchmark/ prefix from the path to run_bench.sh to align with running the commands from within the benchmark directory.
| ./benchmark/run_bench.sh -r 1,10 -c 1,10 -req 1,1024 -resp 1,1024 -rpc_type unary,streaming | |
| ./run_bench.sh -r 1,10 -c 1,10 -req 1,1024 -resp 1,1024 -rpc_type unary,streaming |
| go test -bench=. -benchmem ./benchmark/primitives/ | ||
|
|
||
| # Run a single benchmark: | ||
| go test -bench=BenchmarkCodeStringSwitch -benchmem ./benchmark/primitives/ |
There was a problem hiding this comment.
Remove the benchmark/ prefix from the package paths to align with running the commands from within the benchmark directory.
| go test -bench=. -benchmem ./benchmark/primitives/ | |
| # Run a single benchmark: | |
| go test -bench=BenchmarkCodeStringSwitch -benchmem ./benchmark/primitives/ | |
| go test -bench=. -benchmem ./primitives/ | |
| # Run a single benchmark: | |
| go test -bench=BenchmarkCodeStringSwitch -benchmem ./primitives/ |
|
@luongs3 Thanks a lot for this contribution. Apologies for the delay in responding. Can you have a look at the comments once and address them. |
3 participants
Fixes #8574.
Adds a
README.mdto thebenchmark/directory documenting the commonly used ways to run the benchmarks, and links to it fromCONTRIBUTING.md.What's covered
benchmain— the primary configurable driver, with the canonical example invocation (mirrors the package doc inbenchmark/benchmain/main.go) and the recommended baseline-capture-then-compare workflow usingbenchresult.run_bench.sh— the shell harness that sweeps parameter combinations.go test -benchfor packages likeprimitivesandlatency.The
CONTRIBUTING.mdlink is added under the testing bullet in "Guidelines for Pull Requests", so authors and reviewers can find it when a performance-focused PR needs benchmark results — which is the motivation in the issue.Verification
The documented commands were run against the current tree:
go run benchmark/benchmain/main.go -hlists the flags.go test -bench=BenchmarkCodeStringSwitch -benchmem -run='^$' ./benchmark/primitives/runs and passes.No code changes; docs only.
RELEASE NOTES: n/a