fix(mimir): gate StrongConsistency alert on strong mode - #16477
fix(mimir): gate StrongConsistency alert on strong mode#16477shoemoney wants to merge 2 commits into
Conversation
Fix verified RED->GREEN. MimirStrongConsistencyOffsetMissing fires on default eventual read-consistency at ingest-storage.libsonnet:238
Signed commits report2 of 2 commits between
This repository requires all commits to be signed. See GitHub docs on commit signature verification. |
|
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…ed requests Bugbot: with_offset="true" gate suppressed the alert when offset injection fails completely (all requests missing offsets), the exact case it should catch. Gate on total partition-reader strong-consistency rate instead.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 236f18b. Configure here.
| * 100 > 5 | ||
| ) | ||
| and | ||
| sum by (%(alert_aggregation_labels)s) (rate(cortex_ingest_storage_strong_consistency_requests_total{component="partition-reader"}[%(range)s])) > 0 |
There was a problem hiding this comment.
Alert gate ignores strong consistency
Medium Severity
The and clause no longer requires with_offset="true" samples, so it matches any strong-consistency traffic. When only with_offset="false" requests exist, the ratio is 100% and the total rate is positive, so StrongConsistencyOffsetMissing still fires under default eventual consistency.
Reviewed by Cursor Bugbot for commit 236f18b. Configure here.
1 participant
Fixes MimirStrongConsistencyOffsetMissing firing on default eventual read consistency.
Bug: the alert at operations/mimir-mixin/alerts/ingest-storage.libsonnet computes the ratio of strong-consistency requests missing an offset against the total, but had no gate requiring strong-consistency traffic to exist. In default eventual mode, with no strong-consistency traffic at all, the ratio still evaluated and the alert could fire.
Fix: gate the expression on the total rate of cortex_ingest_storage_strong_consistency_requests_total{component="partition-reader"} being greater than zero, so the alert only evaluates when strong consistency is actually in use.
History: an earlier version of this fix gated on with_offset="true" traffic specifically. Cursor Bugbot correctly flagged that this suppressed the alert during a complete offset-propagation failure (all requests missing an offset), which is the exact case the alert exists to catch. The current version gates on total partition-reader strong-consistency rate instead, so it still fires when offsets are missing on all requests.
Tested with local alert evaluation (RED to GREEN). Single file change, no blast radius beyond ingest-storage.libsonnet.