Remote write 2.0 (part 1/2): move start timestamp, keep writing legacy field for rolling upgrade - #16475
Open
krajorama wants to merge 7 commits into
Open
Remote write 2.0 (part 1/2): move start timestamp, keep writing legacy field for rolling upgrade#16475krajorama wants to merge 7 commits into
krajorama wants to merge 7 commits into
Conversation
…ogram Prometheus's Remote Write 2.0 spec has moved the per-series created_timestamp field to a per-sample/per-histogram start_timestamp field (Sample field 3, Histogram field 17), reserving TimeSeries field 6. Catch Mimir's wire format and internal data model up to this layout. Since each sample/histogram now carries its own start timestamp instead of sharing one per series batch, ingester and block-builder consumption now evaluates each sample/histogram's start timestamp independently (deduplicating consecutive entries that share the same value), rather than only ever acting on the first one in a batch. This intentionally drops support for decoding the old wire-level created_timestamp field; a follow-up commit restores wire compatibility with senders that still use it. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… senders Decode the reserved TimeSeriesRW2 field 6 (formerly created_timestamp) when present on the wire, without resurrecting it as a public Go field, and fan its value out to every Sample and Histogram in that series that doesn't already carry its own StartTimestamp. This restores wire compatibility with Remote Write 2.0 senders that still use the pre-final per-series shape, which the previous commit dropped support for. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds TestDistributor_RW2_StartTimestamp, mirroring TestDistributor_RW2_RC3_CreatedTimestamp but exercising the new per-sample Sample.StartTimestamp field instead of the legacy per-series created_timestamp. Pushes two samples with distinct start timestamps in one request and asserts each independently triggers its own zero sample, which the old per-series field could never express. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TimeSeries.UnmarshalRW2 already fanned the reserved created_timestamp field out to Sample/Histogram.StartTimestamp for external Remote Write 2.0 senders using the pre-final shape. The plain TimeSeries.Unmarshal, used for internal distributor->ingester gRPC and ingest-storage Kafka records, didn't: a not-yet-upgraded Mimir component's created_timestamp would be silently dropped during a rolling upgrade rather than degrading gracefully like the external path. Add the same fan-out there too. Flagged by Cursor Bugbot. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
For a phased rollout across the distributed system, senders must keep writing the reserved TimeSeries/TimeSeriesRW2 field 6 (formerly created_timestamp) alongside the new per-sample/per-histogram StartTimestamp fields, so that not-yet-upgraded readers (internal distributor->ingester gRPC, ingest-storage Kafka records, or external Remote Write 2.0 receivers on the pre-final shape) keep working during the upgrade window. Marshal now derives that legacy value from the first sample's StartTimestamp, or the first histogram's if there are no samples. This is safe for upgraded readers too: TimeSeries.Unmarshal only fans the legacy field out to samples/histograms that don't already carry their own StartTimestamp, so on an upgraded reader the redundant legacy field is always a no-op. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not supporting a not-yet-released version of the Remote Write 2.0 spec wasn't a bug; the CHANGE entry already covers the rollout compatibility behavior. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
krajorama
commented
Aug 25, 2026
| // seriesStartTimestamp tracks the start timestamp of the last data point appended to the | ||
| // corresponding entry in series, parallel to it by index. Used by ctRequiresNewSeries to | ||
| // detect a new counter generation (a different start timestamp) within the same push. | ||
| seriesStartTimestamp []int64 |
Contributor
Author
There was a problem hiding this comment.
note to reviewers: this is needed temporarily because the PreallocTimeseries no longer has space for it. We could reintroduce a private field in PreallocTimeseries, but this way only otlp is impacted which is less surface than all remote write.
krajorama
marked this pull request as ready for review
August 25, 2026 18:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
This is part 1 of 2 for a phased rollout of the Remote Write 2.0 start-timestamp change across the distributed Mimir system (see #16471 for part 2, which will land only once this has fully rolled out).
Prometheus's Remote Write 2.0 spec moved the per-series
created_timestampfield to a per-sample/per-histogramstart_timestampfield (Samplefield 3,Histogramfield 17), reservingTimeSeriesfield 6.mimirpbhelpers). Since each sample/histogram now carries its own start timestamp, ingestion evaluates each one's start timestamp independently (deduplicating consecutive entries sharing the same value) instead of only ever acting on the first one in a batch. The OTLP appender keeps splitting a series into a new wireTimeSeriesentry whenever the start timestamp changes (unlike part 2), so each entry it produces has exactly one start timestamp to report on the legacy field below.Sample/Histogramin that series that doesn't already carry its ownStartTimestamp. This covers both external Remote Write 2.0 senders using the pre-final shape, and internal distributor→ingester gRPC / ingest-storage Kafka records from a not-yet-upgraded component.TimeSeries/TimeSeriesRW2marshalling now also writes the reserved per-series field 6, derived from the first sample'sStartTimestamp, or the first histogram's if there are no samples. This lets not-yet-upgraded readers (old ingesters, old internal tooling) keep working correctly throughout the rollout, since they only understand the old per-series field. This is a no-op for upgraded readers, sinceUnmarshalonly fans the legacy value into a sample/histogram that doesn't already have its ownStartTimestamp.Rollout plan
Test plan
pkg/mimirpb,pkg/ingester,pkg/blockbuilder,pkg/distributor(includingotlpappender), andpkg/continuoustestpkg/mimirpb/compat_rw2_test.go:Sample/Histogram.StartTimestampround-trips over RW1 and RW2, legacy per-seriescreated_timestampfan-out (both the RW2-specific and the plain internal decode path), andTestMarshalLegacyCreatedTimestampverifying the new marshal-side legacy field is written correctly (and is a no-op on round-trip through an upgraded reader)TestDistributor_RW2_StartTimestamp: pushes two samples with distinct start timestamps in one request and asserts each independently triggers its own zero samplemake check-protos-equivalent regeneration verified idempotentgo build ./...and broadgo testsweep pass (aside from known pre-existing/environmental flakes unrelated to this change)