A benchmarking tool for sizing self-managed vLLM inference on Amazon EKS. Concurrency sweeps with a correctness gate, a protocol lock on every result, and a serving recipe with a runbook for every model, precision and instance type.
Quickstart | Recipes | Runbooks | Guides | Architecture | Reference | FAQs
Each capability names the tool it is built on. This repo does not reimplement what already exists; it supplies the methodology around it.
| Capability | What it does | Why it matters |
|---|---|---|
| ✅ Accuracy gate | Compares the live server's output against a pinned reference before any measurement runs, scoring a pinned GSM8K subset committed to this repo | A misconfigured server aborts the run instead of producing a fast, wrong result. The gate fails closed, so a missing reference stops the run rather than passing it |
| 🔒 Protocol lock and provenance | Opens every result file with the conditions the run was taken under, and closes it with the serving pod's argv, NCCL transport, KV sizing and GPU topology | A number cannot drift away from what produced it, so a result stays comparable against one taken a year later |
| 📝 Pinned prompt profiles | Ships the literal prompt text as committed JSON, generated deterministically, with per-model wire token counts measured offline through Hugging Face Transformers | Every model receives byte-identical input, so a difference between two models belongs to the models. Input size is recorded in every result rather than inferred from a generator |
| 🧪 Manifest validation | Re-runs each recipe's script with vLLM stubbed out and compares the argv that arrives against the flags the recipe is written to pass | Catches a recipe that applies cleanly but serves the wrong configuration, before GPU time is spent trusting it |
| ☸️ In-cluster measurement | Runs the harness as a Job beside the server, reaching it over cluster DNS | A port-forward adds tunnel setup to every TTFT sample, so the harness refuses to report first-token latency through one |
| 📈 Closed-loop concurrency sweeps | Sweeps 1 to 512 concurrent requests on this repo's own aiohttp client, reporting tokens/sec, req/sec and TTFT, TPOT and ITL across three reps with medians | Converts a latency SLA into a node count, which is the sizing decision |
| 🎯 Open-loop SLA verification | Issues Poisson arrivals at a target QPS through MLPerf LoadGen, with direct issue-rate accounting | Answers whether the SLA holds at a given arrival rate, a different question from the sweeps, so the two never share a table |
| 🔀 Cross-client calibration | Reruns the same workload through NVIDIA AIPerf behind the same protocol seam, and replays committed agentic traces with real think-time gaps | Two independent clients on one server show whether a figure belongs to the server or to the client that measured it |
| 📦 Weight staging | One Job per model and precision, applied once per filesystem or bucket, that stages the checkpoint to your S3 bucket or the shared claim | Recipes wait for weights rather than fetching them, so a serving run is not also a download, and a partial checkpoint fails before the GPU is booked |
| 🗜️ Quantization pipeline | Produces FP8, INT8 and INT4-AWQ checkpoints with NVIDIA Model Optimizer, as a GPU Job in your own cluster | Public quantized checkpoints do not exist for every model, and converting one needs more memory than serving the result afterwards |
| ⚙️ Serving-stack coverage | Covers prefix caching, KV-cache precision, FlashInfer and Triton attention backends, and tensor-parallel serving up to eight GPUs per node as Deployments; multi-node pipeline parallelism over EFA is covered in experiments | The serving configuration is the object of study, so the knobs that move a number are already wired up |
| 🔬 Profiling and interconnect studies | Captures Nsight Systems traces, and isolates the interconnect with nccl-tests | Explains a ceiling that a sweep can only locate. A trace has already overturned a conclusion this repo had published |
| 📚 Runbooks, not snippets | Pairs every serving recipe with capacity sizing, prerequisites, verified steps and tear-down, from single-GPU g7e.2xlarge to 8-GPU p6-b200.48xlarge |
The starting point is a working deployment rather than a blank manifest |
- Install Python 3.10+
- Create an EKS cluster with Karpenter, Karpenter discovery tags and node role
- Get a Hugging Face token (read-only access)
# 1. Clone this repository
git clone https://github.com/aws-samples/sample-eks-inference-benchmarking-tool.git
cd sample-eks-inference-benchmarking-tool
# 2. Set up a virtual environment
python3 -m venv .venv
source .venv/bin/activate
# 3. Install the CLI
python3 -m pip install --upgrade pip setuptools wheel
pip install -e '.[tokenizer]'
# 4. Verify the installation
eks-bench --helpRecipes without a published quantized checkpoint build the quantizer image in-cluster with kaniko. The runbook step does this and needs no local Docker. To build on x86 Docker instead, run docker login and then ./scripts/push-image.sh.
Choose a serving runbook from vLLM recipes. Each one stands on its own: it sets its environment, stages weights with a script that fails on an incomplete checkpoint, and deploys a server whose model and flags are checked before use. Every benchmark runs behind an accuracy gate, follows an identical --dry-run, and records its own provenance. Run the steps in order.
Quantization reduces the size of the model weights, so a model can fit on fewer or smaller GPUs, allowing more KV cache headroom to run your workloads.
Some runbooks stage published quantized checkpoints with Jobs from model-jobs/. Recipes without a usable published checkpoint are quantized with NVIDIA Model Optimizer. Each of those recipes has a quantize Job and script in model-jobs/, and its runbook's instructions run it.
Each runbook below contains the prerequisites to run each benchmark, the steps to run the benchmark, and tear down steps. Choose a model below to find available recipes, with links to each runbook.
Follow the vllm-recipes/README.md for an index of all runbooks, with instance, GPU count, parallelism, workload kind and serving image per recipe.
NCCL is NVIDIA's collective communications library, and it moves tensors between GPUs on every step of a tensor-parallel forward pass. These studies measure the interconnect rather than the model: microbenchmarks with no model or vLLM stack at all, and serving arms that run vLLM like a recipe but exist to compare NCCL_P2P_LEVEL settings rather than to measure serving performance.
Follow the experiments/README for experiments and studies, including the PCIe-switch-vs-host-copy study, and the recipes that measure the interconnect.
The benchmark harnesses in this project tell you how slow. An Nsight Systems trace tells you why: how step time splits across GEMM, the NCCL collective, attention, softmax, sampling and scheduling gaps. Reach for it when a sweep shows a ceiling and nothing explains it. A trace has already overturned one published conclusion: it showed the NCCL collective consuming a substantial share of GPU kernel time after the P2P study above had called it a small, well-overlapped fraction of step time.
Follow the Profile GPUs with Nsight Systems runbook for the prerequisites and the capture itself, an eight-step in-cluster walkthrough.
Use the Reference guide to learn about the measurement CLI: the measure and measure-local commands, their flags, the environment variables a run sets, the benchmark protocols, and the rules for which results may be compared with which. It includes Which benchmark to run, which covers what each protocol measures, what tables it returns and what it does not report.
Run eks-bench --help for the command list, and eks-bench <command> --help for a command's flags. Click and Rich stay in the operator CLI and do not remain on the measurement path after dispatch: each command execs app/bench/src/main.py, which depends only on the standard library and aiohttp.
Use the FAQs guide for scope questions (does it cover training, Inferentia, non-EKS deployments), what data leaves your environment, and how this repo compares to MLPerf, vllm bench serve, AIPerf, FMBench and general-purpose load testers.
See CONTRIBUTING for more information.
This project is licensed under the Apache-2.0 License.