Skip to content

*: revert gRPC stream operation duration metrics (#11048) - #11049

Merged
ti-chi-bot[bot] merged 1 commit into
tikv:release-nextgen-202603from
ti-chi-bot:cherry-pick-11048-to-release-nextgen-202603
Jul 30, 2026
Merged

*: revert gRPC stream operation duration metrics (#11048)#11049
ti-chi-bot[bot] merged 1 commit into
tikv:release-nextgen-202603from
ti-chi-bot:cherry-pick-11048-to-release-nextgen-202603

Conversation

@ti-chi-bot

Copy link
Copy Markdown
Member

This is an automated cherry-pick of #11048

What problem does this PR solve?

Issue Number: close #11040

grpcutil.NewMetricsStream registers histogram children labeled by peer IP with no lifecycle cleanup. Long-running PD/MCS processes accumulate unbounded label series (observed ~840k series from ~18k historical IPs), inflating memory and /metrics scrape cost until scrapes fail.

What is changed and how does it work?

Revert the gRPC stream operation duration metrics introduced in #10254 and the dependent Grafana stream-send panels from #10339. This removes `grpcutil.MetricsStream` and the peer-IP-labeled `grpc_stream_operation_duration_seconds` histograms from PD and MCS services, eliminating the unbounded cardinality leak until a safer replacement lands.

Check List

Tests

  • No code

Related changes

  • Need to cherry-pick to the release branch

Release note

Revert gRPC stream operation duration metrics to fix unbounded peer-IP label cardinality.

Summary by CodeRabbit

  • Bug Fixes
    • Removed gRPC stream send-duration metrics from server endpoints and Grafana dashboards.
    • Simplified streaming operations by eliminating the associated instrumentation layer.
    • Removed the unsupported gRPC receive error and related utility behavior.
    • Existing request handling and service functionality remain unchanged.

@ti-chi-bot ti-chi-bot added the dco-signoff: yes Indicates the PR's author has signed the dco. label Jul 24, 2026
@ti-chi-bot ti-chi-bot Bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jul 24, 2026
@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-nextgen-202603 labels Jul 24, 2026
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 262dcc00-0037-4a2a-81a8-7fb313beed76

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 24, 2026
@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jul 24, 2026
@JmPotato
JmPotato force-pushed the cherry-pick-11048-to-release-nextgen-202603 branch from 3b76282 to e32c1cc Compare July 24, 2026 10:06
@ti-chi-bot ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 24, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JmPotato, lhy1024

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 24, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-07-24 09:58:37.51829552 +0000 UTC m=+1571703.554390576: ☑️ agreed by JmPotato.
  • 2026-07-24 10:13:48.198628681 +0000 UTC m=+1572614.234723727: ☑️ agreed by lhy1024.

@JmPotato

Copy link
Copy Markdown
Member

/retest

2 similar comments
@JmPotato

Copy link
Copy Markdown
Member

/retest

@JmPotato

Copy link
Copy Markdown
Member

/retest

close tikv#11040\n\nRevert the gRPC stream operation duration metrics introduced in tikv#10254 and the dependent Grafana stream-send panels from tikv#10339. This removes `grpcutil.MetricsStream` and the peer-IP-labeled `grpc_stream_operation_duration_seconds` histograms from PD and MCS services, eliminating the unbounded cardinality leak until a safer replacement lands.\n\nSigned-off-by: JmPotato <github@ipotato.me>

(cherry picked from commit 070828c)
Signed-off-by: JmPotato <github@ipotato.me>
@JmPotato
JmPotato force-pushed the cherry-pick-11048-to-release-nextgen-202603 branch from e32c1cc to 026367b Compare July 29, 2026 08:02
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.16%. Comparing base (84a02ea) to head (026367b).
⚠️ Report is 1 commits behind head on release-nextgen-202603.

Additional details and impacted files
@@                    Coverage Diff                     @@
##           release-nextgen-202603   #11049      +/-   ##
==========================================================
- Coverage                   79.19%   79.16%   -0.04%     
==========================================================
  Files                         532      530       -2     
  Lines                       72818    72726      -92     
==========================================================
- Hits                        57668    57570      -98     
- Misses                      11114    11122       +8     
+ Partials                     4036     4034       -2     
Flag Coverage Δ
unittests 79.16% <100.00%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JmPotato

Copy link
Copy Markdown
Member

/retest

@JmPotato

Copy link
Copy Markdown
Member

/test pull-unit-test-next-gen-2

@ti-chi-bot
ti-chi-bot Bot merged commit 8b6739b into tikv:release-nextgen-202603 Jul 30, 2026
26 of 27 checks passed
@ti-chi-bot
ti-chi-bot Bot deleted the cherry-pick-11048-to-release-nextgen-202603 branch July 30, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. type/cherry-pick-for-release-nextgen-202603

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants