tools: add duplicate-samples-generator - #16465
Open
dmatth1 wants to merge 2 commits into
Open
Conversation
Adds a tool that sends remote-write requests containing duplicate-timestamp samples and asserts how Mimir accounts for them, covering the flows enumerated in the Dropped Samples design doc. Each flow asserts that the expected reason on cortex_discarded_samples_total gains exactly one count per duplicate, that no other reason moved, and that cortex_ingester_ingested_samples_total grows by the non-duplicate samples only. Duplicates are produced in six shapes, with and without a value conflict, from float samples or native histograms. Shapes that keep the duplicate inside one TimeSeries object are collapsed by the distributor and are sent to -address; the rest have to reach the ingester and are sent to -ingester-address over its Push RPC, since a distributor would otherwise collapse them first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4e3971f. Configure here.
… check - flag.Parse is blocklisted by faillint; use dskit flagext.ParseFlagsWithoutArguments. - Attributed discards always read zero: scrape keys the metric by reason, while report read the bare name. cortex_discarded_attributed_samples_total also names the tenant "tenant" rather than "user", so the tenant filter skipped it. - -series-per-request 1 is valid for shapes that send the duplicate in its own request; only same-request and ooo-same-request need room for a pair. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Adds tool to remote-write requests containing duplicate-timestamp samples and assert how Mimir accounts for them.
What it asserts
3 things per flow:
cortex_discarded_samples_totalgains exactly one count per duplicate.cortex_ingester_ingested_samples_totalgrows by the non-duplicate samples only.Validation
Run against a local Mimir, both before and after the ingester fixes in #16347 and grafana/mimir-prometheus#1259.
Before the fix - 14 of 24 flows fail (the ones expected):
Nothing recorded as discarded and 1000 counted as ingested where only 500 were stored.
After the fix — all 24 pass:
The same flow now reads
want 500 / observed 500on both assertions.Also run against a dev cell through the gateway at replication factor 3, scraping the distributor and all three ingesters: all 24 flows passed, and the stored values were confirmed to be the first-seen ones.