Skip to content

Add LLM benchmark analysis and Discord reporting - #5608

Open
bradleyshep wants to merge 84 commits into
masterfrom
bradley/llm-analysis-expansion
Open

Add LLM benchmark analysis and Discord reporting#5608
bradleyshep wants to merge 84 commits into
masterfrom
bradley/llm-analysis-expansion

Conversation

@bradleyshep

@bradleyshep bradleyshep commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Note: based on #5583

Description of Changes

  • Generates failure analysis after each LLM benchmark run
  • Publishes per-model reports as workflow artifacts and a GitHub Actions summary
  • Posts a formatted Discord summary for scheduled runs or optional manual runs
  • Adds configurable low, medium, or high reasoning, defaulting to medium
  • Reports provider failures when no model output is available
  • Uses the LLM_BENCHMARK_DISCORD_WEBHOOK_URL repository secret for Discord posting

API and ABI breaking changes

  • None

Expected complexity level and risk

  • 2/5 — Low risk
  • Changes are isolated to the LLM benchmark tooling and its GitHub Actions workflow
  • No impact on the product runtime or public APIs

Testing

  • Ran benchmark analysis unit tests
  • Ran Discord summary tests
  • Ran formatting and Clippy checks
  • Verified the default OpenRouter models accept medium reasoning
  • Configure LLM_BENCHMARK_DISCORD_WEBHOOK_URL with a webhook
  • Run the periodic workflow manually and confirm the workflow and Discord summaries

bradleyshep added 30 commits July 17, 2026 16:57
@bradleyshep
llm-benchmark: add normalized table evals
8285dac
@bradleyshep
llm-benchmark: add behavioral table evals
e51e287
@bradleyshep
llm-benchmark: add reducer behavior evals
ba223b3
@bradleyshep
llm-benchmark: add scheduled batch deletion eval
0d3f74c
@bradleyshep
llm-benchmark: add query builder view evals
fb2f938
@bradleyshep
llm-benchmark: add procedural view evals
644340e
@bradleyshep
llm-benchmark: complete view eval coverage
2f647d6
@bradleyshep
llm-benchmark: add auth and lifecycle evals
ba06e21
@bradleyshep
llm-benchmark: add procedure and http evals
8c510ff
@bradleyshep
llm-benchmark: add migration-aware default eval
ce87575
@bradleyshep
llm-benchmark: add migration and RLS evals
633f6d7
@bradleyshep
Support stable local LLM benchmark builds
8106177
@bradleyshep
Retry silent LLM benchmark publish failures
6c3139f
@bradleyshep
Make local LLM benchmark logs collision proof
104fafd
@bradleyshep
Include existing modules in migration eval prompts
25c8b31
@bradleyshep
Make advanced LLM evals self-contained
401b206
@bradleyshep
Document advanced TypeScript server APIs
f49cc64
@bradleyshep
Validate LLM benchmark goldens behaviorally
43c0330
@bradleyshep
Generalize advanced TypeScript server guidance
6d06baa
@bradleyshep
Clarify advanced view eval requirements
9a775e5
@bradleyshep
Document TypeScript server API edge cases
2f6bb29
@bradleyshep
Grade only specified HTTP response headers
b3f4bbc
@bradleyshep
Bound LLM benchmark scorer calls
4a084de
@bradleyshep
Document advanced Rust and C# server APIs
239ebbe
@bradleyshep
Refine TypeScript eval guidance and prompts
fd97271
@bradleyshep
Align new eval prompts with benchmark format
87ee8c8
@bradleyshep
Retry transient HTTP eval calls
5567d91
@bradleyshep
Clarify ambiguous eval task contracts
3c250b9
@bradleyshep
Fix reducer schema parity extraction
4a1252e
@bradleyshep
Strengthen view eval result grading
081432f
bradleyshep added 24 commits July 21, 2026 10:57
@bradleyshep
Clarify C# connection ID encoding eval
219d866
@bradleyshep
Merge remote-tracking branch 'origin/bradley/fix-csharp-llm-bench' in…
b58c1fa 
…to bradley/new-llm-benchmark-evals-0717

# Conflicts:
#	tools/xtask-llm-benchmark/src/bench/publishers.rs
#	tools/xtask-llm-benchmark/src/bench/runner.rs
#	tools/xtask-llm-benchmark/src/bin/llm_benchmark.rs
@bradleyshep
Harden LLM benchmark validation and cleanup
e96a9fc
@bradleyshep
Clarify HTTP APIs in server benchmark skills
9bcdc72
@bradleyshep
Merge branch 'master' into bradley/new-llm-benchmark-evals-0717
5c5eb8f 
# Conflicts:
#	skills/typescript-server/SKILL.md
@bradleyshep
Clarify TypeScript benchmark guidance
438f71c
@bradleyshep
Harden benchmark regression tests
5aac866
@bradleyshep
Remove bias
bdbeb96
@bradleyshep
remove bias
21125cb
@bradleyshep
cleanup
a4a4a3a
@bradleyshep
Use workspace CLI for LLM benchmark runs
07ad955
@bradleyshep
Parallelize periodic benchmarks by language
173fa66
@bradleyshep
Merge branch 'master' into bradley/new-llm-benchmark-evals-0717
02e1781
@bradleyshep
Keep parallel Rust benchmark builds isolated
3483bd6
@bradleyshep
Refine TypeScript eval guidance and randomness scoring
30dbf5a
@bradleyshep
updates
3571338
@bradleyshep
Merge branch 'master' into bradley/new-llm-benchmark-evals-0717
808baa4
@bradleyshep
lints
5b46b13
@bradleyshep
analysis + post to discord
0db8781
@bradleyshep
Update llm-benchmark-periodic.yml
c3e0fb6
@bradleyshep
discord summary script
c018a44
@bradleyshep
reasoning arg
7c265e5
@bradleyshep
Merge branch 'master' into bradley/llm-analysis-expansion
48f890c
@bradleyshep
polish
ef59cc2
@bradleyshep
bradleyshep requested a review from aasoni July 28, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aasoni aasoni Awaiting requested review from aasoni
@bfops bfops Awaiting requested review from bfops bfops is a code owner
@jdetter jdetter Awaiting requested review from jdetter jdetter is a code owner
@cloutiertyler cloutiertyler Awaiting requested review from cloutiertyler cloutiertyler is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bradleyshep