Skip to content

Remote write 2.0 (part 1/2): move start timestamp, keep writing legacy field for rolling upgrade - #16475

Open
krajorama wants to merge 7 commits into
mainfrom
krajo/rw2-start-timestamp-part1
Open

Remote write 2.0 (part 1/2): move start timestamp, keep writing legacy field for rolling upgrade#16475
krajorama wants to merge 7 commits into
mainfrom
krajo/rw2-start-timestamp-part1

Conversation

@krajorama

Copy link
Copy Markdown
Contributor

What this does

This is part 1 of 2 for a phased rollout of the Remote Write 2.0 start-timestamp change across the distributed Mimir system (see #16471 for part 2, which will land only once this has fully rolled out).

Prometheus's Remote Write 2.0 spec moved the per-series created_timestamp field to a per-sample/per-histogram start_timestamp field (Sample field 3, Histogram field 17), reserving TimeSeries field 6.

  1. Move start timestamp from TimeSeries to Sample/Histogram — adopts the new schema: moves the field, reserves the old one, and adapts every internal consumer (ingester, block-builder, mimirpb helpers). Since each sample/histogram now carries its own start timestamp, ingestion evaluates each one's start timestamp independently (deduplicating consecutive entries sharing the same value) instead of only ever acting on the first one in a batch. The OTLP appender keeps splitting a series into a new wire TimeSeries entry whenever the start timestamp changes (unlike part 2), so each entry it produces has exactly one start timestamp to report on the legacy field below.
  2. Restore compatibility with legacy created_timestamp senders — decodes the reserved field 6 when present on the wire (without resurrecting it as a public Go field) and fans it out to every Sample/Histogram in that series that doesn't already carry its own StartTimestamp. This covers both external Remote Write 2.0 senders using the pre-final shape, and internal distributor→ingester gRPC / ingest-storage Kafka records from a not-yet-upgraded component.
  3. Marshal the legacy field for not-yet-upgraded readers — the most important part of this PR: TimeSeries/TimeSeriesRW2 marshalling now also writes the reserved per-series field 6, derived from the first sample's StartTimestamp, or the first histogram's if there are no samples. This lets not-yet-upgraded readers (old ingesters, old internal tooling) keep working correctly throughout the rollout, since they only understand the old per-series field. This is a no-op for upgraded readers, since Unmarshal only fans the legacy value into a sample/histogram that doesn't already have its own StartTimestamp.

Rollout plan

  • This PR (part 1) can be deployed safely to any subset of the fleet in any order: old and new binaries remain interoperable, since every write carries both the modern per-sample fields and the legacy per-series fallback.
  • Once the whole fleet (and, for the ingest-storage architecture, Kafka retention) has rolled forward past this change, part 2 removes the legacy marshalling and the OTLP appender's start-timestamp-based series splitting, since they're no longer needed.

Test plan

  • Unit tests updated/added across pkg/mimirpb, pkg/ingester, pkg/blockbuilder, pkg/distributor (including otlpappender), and pkg/continuoustest
  • New tests in pkg/mimirpb/compat_rw2_test.go: Sample/Histogram.StartTimestamp round-trips over RW1 and RW2, legacy per-series created_timestamp fan-out (both the RW2-specific and the plain internal decode path), and TestMarshalLegacyCreatedTimestamp verifying the new marshal-side legacy field is written correctly (and is a no-op on round-trip through an upgraded reader)
  • New integration test TestDistributor_RW2_StartTimestamp: pushes two samples with distinct start timestamps in one request and asserts each independently triggers its own zero sample
  • make check-protos-equivalent regeneration verified idempotent
  • go build ./... and broad go test sweep pass (aside from known pre-existing/environmental flakes unrelated to this change)

krajorama and others added 7 commits August 25, 2026 19:18
…ogram

Prometheus's Remote Write 2.0 spec has moved the per-series
created_timestamp field to a per-sample/per-histogram start_timestamp
field (Sample field 3, Histogram field 17), reserving TimeSeries field
6. Catch Mimir's wire format and internal data model up to this
layout.

Since each sample/histogram now carries its own start timestamp
instead of sharing one per series batch, ingester and block-builder
consumption now evaluates each sample/histogram's start timestamp
independently (deduplicating consecutive entries that share the same
value), rather than only ever acting on the first one in a batch.

This intentionally drops support for decoding the old wire-level
created_timestamp field; a follow-up commit restores wire
compatibility with senders that still use it.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… senders

Decode the reserved TimeSeriesRW2 field 6 (formerly created_timestamp)
when present on the wire, without resurrecting it as a public Go
field, and fan its value out to every Sample and Histogram in that
series that doesn't already carry its own StartTimestamp. This
restores wire compatibility with Remote Write 2.0 senders that still
use the pre-final per-series shape, which the previous commit dropped
support for.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds TestDistributor_RW2_StartTimestamp, mirroring
TestDistributor_RW2_RC3_CreatedTimestamp but exercising the new
per-sample Sample.StartTimestamp field instead of the legacy
per-series created_timestamp. Pushes two samples with distinct start
timestamps in one request and asserts each independently triggers its
own zero sample, which the old per-series field could never express.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TimeSeries.UnmarshalRW2 already fanned the reserved created_timestamp
field out to Sample/Histogram.StartTimestamp for external Remote
Write 2.0 senders using the pre-final shape. The plain
TimeSeries.Unmarshal, used for internal distributor->ingester gRPC
and ingest-storage Kafka records, didn't: a not-yet-upgraded Mimir
component's created_timestamp would be silently dropped during a
rolling upgrade rather than degrading gracefully like the external
path. Add the same fan-out there too.

Flagged by Cursor Bugbot.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
For a phased rollout across the distributed system, senders must keep
writing the reserved TimeSeries/TimeSeriesRW2 field 6 (formerly
created_timestamp) alongside the new per-sample/per-histogram
StartTimestamp fields, so that not-yet-upgraded readers (internal
distributor->ingester gRPC, ingest-storage Kafka records, or external
Remote Write 2.0 receivers on the pre-final shape) keep working during
the upgrade window. Marshal now derives that legacy value from the
first sample's StartTimestamp, or the first histogram's if there are
no samples.

This is safe for upgraded readers too: TimeSeries.Unmarshal only fans
the legacy field out to samples/histograms that don't already carry
their own StartTimestamp, so on an upgraded reader the redundant
legacy field is always a no-op.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not supporting a not-yet-released version of the Remote Write 2.0
spec wasn't a bug; the CHANGE entry already covers the rollout
compatibility behavior.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
// seriesStartTimestamp tracks the start timestamp of the last data point appended to the
// corresponding entry in series, parallel to it by index. Used by ctRequiresNewSeries to
// detect a new counter generation (a different start timestamp) within the same push.
seriesStartTimestamp []int64

@krajorama krajorama Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note to reviewers: this is needed temporarily because the PreallocTimeseries no longer has space for it. We could reintroduce a private field in PreallocTimeseries, but this way only otlp is impacted which is less surface than all remote write.

@krajorama
krajorama marked this pull request as ready for review August 25, 2026 18:18
@krajorama
krajorama requested a review from a team as a code owner August 25, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant