Skip to content

Reduce BenchmarkValidation overhead by caching safe-output field order#44090

Draft
pelikhan with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-benchmark-validation-performance
Draft

Reduce BenchmarkValidation overhead by caching safe-output field order#44090
pelikhan with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-benchmark-validation-performance

Conversation

Copilot AI commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

BenchmarkValidation regressed due to repeated work in validation hot paths. The dominant cost was re-sorting safe-output field mappings on each validation/replay pass, inflating CPU and allocations.

  • Root-cause reduction in validation path

    • Added a sync.Once-backed cache for sorted safeOutputFieldMapping keys in samples_validation.go.
    • Replaced per-call sorting in validateSafeOutputsSamples with cached lookup.
  • Keep replay path consistent

    • Updated collectSampleEntries in samples_replay.go to use the same cached field-order helper, preserving deterministic ordering while removing duplicate sort work.
  • Targeted coverage

    • Added TestGetSortedSafeOutputFieldNames_MatchesSortedKeys to confirm:
      • cached ordering matches canonical sorted keys
      • repeated calls reuse the same backing slice behavior (guarded for non-empty slices)
var (
	sortedSafeOutputFieldNamesOnce sync.Once
	sortedSafeOutputFieldNames     []string
)

func getSortedSafeOutputFieldNames() []string {
	sortedSafeOutputFieldNamesOnce.Do(func() {
		sortedSafeOutputFieldNames = sliceutil.SortedKeys(safeOutputFieldMapping)
	})
	return sortedSafeOutputFieldNames
}

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Hey @Copilot 👋 — great to see a fix being kicked off for the BenchmarkValidation performance regression (25.4% slower, 38,168 ns/op vs historical 30,437 ns/op). This is squarely the right thing to work on.

This PR is still a fresh draft with no code changes yet, so a couple of things to keep in mind as it takes shape:

  • Add a description — once a plan or approach is decided, update the PR body to explain what is being changed and why (e.g. which code path was identified as the hotspot, what optimization was applied).
  • Include benchmark tests — since this is a performance regression fix, please include before/after go test -bench=BenchmarkValidation output in the PR body or as a comment, and consider adding or updating the benchmark itself so the regression can't silently reappear.

When ready, here's a prompt you can use to drive the implementation:

Fix the 25.4% performance regression in BenchmarkValidation (currently 38,168 ns/op vs historical baseline of ~30,437 ns/op).

1. Run `make bench-memory` and `go test -bench=BenchmarkValidation -benchmem -count=5 ./...` to reproduce and measure the current state.
2. Use `go tool pprof` to identify the hotspot introduced by recent changes to validation logic.
3. Apply the minimal targeted fix to bring ns/op back to or below the 30,437 baseline.
4. Confirm with `benchstat` comparing before/after runs (at least 5 iterations each).
5. Update the PR body with: (a) root cause identified, (b) fix applied, (c) before/after benchmark numbers.

Generated by ✅ Contribution Check · 122 AIC · ⌖ 17.8 AIC · ⊞ 6.2K ·

Copilot AI and others added 3 commits July 7, 2026 17:45
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix performance regression in BenchmarkValidation Reduce BenchmarkValidation overhead by caching safe-output field order Jul 7, 2026
Copilot AI requested a review from pelikhan July 7, 2026 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[performance] Regression in BenchmarkValidation: 25.4% slower

2 participants