Skip to content

[Bench] Add external baselines for common ops - #1810

Draft
superAngGao wants to merge 2 commits into
tile-ai:mainfrom
superAngGao:bench/external-baselines-1806
Draft

[Bench] Add external baselines for common ops#1810
superAngGao wants to merge 2 commits into
tile-ai:mainfrom
superAngGao:bench/external-baselines-1806

Conversation

@superAngGao

@superAngGao superAngGao commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Draft PR for #1806.

  • Add external baseline tags for common-op benchmarks: torch_inductor, torch_cudnn, torch_cufft, torch_native, and flashinfer.
  • Add a shared torch_inductor_baseline() helper for benchmark-local PyTorch baselines compiled with torch.compile(..., fullgraph=True).
  • Add FlashInfer baselines for RMSNorm, fused add RMSNorm, and semantic-equivalent RoPE variants.
  • Keep optional third-party baselines local to benchmark files and preserve existing roofline reporting.
  • Fix nightly report coloring to use the strongest competitive baseline when multiple baselines are present.

Validation

Official runner image: ghcr.io/tile-ai/tileops-runner:65dbc98-torch2.10.

  • compileall and ruff check on changed files.
  • pytest -q tests/test_nightly_report.py benchmarks/tests/test_benchmark_base.py --tb=short -> 19 passed.
  • RoPE/Pool benchmark collect-only -> 52 cases collected.
  • Targeted manifest checks for RoPE, RMSNorm, Conv/Pool, and FFT passed with advisory warnings only.
  • TorchInductor and FlashInfer smoke checks passed.

Closes #1806.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@github-actions github-actions Bot added the bench Benchmark updates label Jul 30, 2026
@superAngGao

Copy link
Copy Markdown
Collaborator Author

Thanks for the careful review. I pushed a follow-up commit (1a28c9d) addressing all three points.

  • Fixed RoPE FlashInfer semantics: cached FlashInfer baselines now receive variant-specific cos/sin caches. YaRN uses the benchmark-local YaRN interpolation formula, and the benchmark also checks FlashInfer output against the benchmark-local semantic reference once before timing. I also tightened the torch reference path so Neox, Non-Neox, Llama 3.1, YaRN, and LongRoPE all use semantic-equivalent references instead of geometry-only rotation.
  • Renamed pooling baselines from torch_cudnn to torch_native, matching the kernels observed in the official runner. Convolution and BatchNorm remain tagged torch_cudnn.
  • Fixed nightly report coloring for multi-baseline configs. Since ratio = baseline_latency / tileops_latency, the strongest competitor is the smallest ratio, not the largest. Added a small regression test for the FlashInfer=0.5 / Inductor=2.0 case.

Validation in ghcr.io/tile-ai/tileops-runner:65dbc98-torch2.10:

  • compileall for the changed files
  • pytest -q tests/test_nightly_report.py benchmarks/tests/test_benchmark_base.py --tb=short -> 19 passed
  • pytest --collect-only -q benchmarks/ops/bench_rope.py benchmarks/ops/bench_pool.py --tb=short -> 52 benchmark cases collected
  • ruff check on the changed files
  • validate_manifest.py --check-op RopeYarnOp
  • validate_manifest.py --check-op AvgPool2dFwdOp
  • FlashInfer RoPE semantic smoke: Neox, Non-Neox, Llama 3.1, YaRN, and LongRoPE all matched the benchmark-local reference within fp16 tolerance; YaRN/Llama/LongRoPE local tables also matched the production formula helpers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bench Benchmark updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bench] Add external baselines for roofline-only common ops

1 participant