Skip to content

executor, store/copr: batch full-sampling Analyze requests#70055

Draft
0xTars wants to merge 3 commits into
pingcap:masterfrom
0xTars:copper/analyze-batch-merge-v2
Draft

executor, store/copr: batch full-sampling Analyze requests#70055
0xTars wants to merge 3 commits into
pingcap:masterfrom
0xTars:copper/analyze-batch-merge-v2

Conversation

@0xTars

@0xTars 0xTars commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: ref #70059

Problem Summary:

Full-sampling Analyze sends many region coprocessor requests. Store batching can reduce RPC overhead, but the client and store need an explicit merged-result contract, complete per-task resource accounting, and a concurrency budget that does not multiply the existing Analyze DistSQL or system-process scan limit by the batch width.

This replaces #69686 because its source branch cannot be updated. Issue #70059 tracks this batching design; #67449 is related sample-based NDV work, not the design source for this PR.

What changed and how does it work?

  • Group compatible full-sampling Analyze region tasks by store and send them through StoreBatchTasks.
  • Opt in to the kvproto merged-result contract from coprocessor: negotiate merged batch task data kvproto#1511 and consume each child acknowledgment by its unique task ID.
  • Use resourcecontrol: settle batched cop tasks' execution details tikv/client-go#2032 so scan bytes and KV CPU from every attached batch response participate in resource-control settlement.
  • Derive batching from the already-resolved Analyze scan budget C: width = min(C, 5), outer concurrency C / width, and width - 1 child tasks. Therefore total logical scan concurrency remains at most C for both manual Analyze and restricted/system Analyze.
  • Keep the package test count at the repository limit by grouping the two related batch-response scenarios into one behavior suite; update the existing executor/store test case maps.

Related replacement chain:

Check List

Tests

  • Unit test
    • ./tools/check/failpoint-go-test.sh pkg/executor -run '^TestAnalyzeBatchScanBudget$' -count=1
    • ./tools/check/failpoint-go-test.sh pkg/store/copr -run '^TestHandleBatchCopResponse$' -count=1
    • ./tools/check/failpoint-go-test.sh pkg/store/copr/copr_test -run '^TestBuildCopIteratorWithBatchStoreCopr$' -count=1
    • make lint
    • make bazel_prepare (zero generated diff)
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Reduce full-sampling ANALYZE RPC overhead while keeping total logical scan concurrency within the Analyze scan budget.

@ti-chi-bot ti-chi-bot Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jul 24, 2026
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7fbe3a46-0d8a-4c1d-b612-f6378c4cf420

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot

ti-chi-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

Hi @0xTars. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot Bot added contribution This PR is from a community contributor. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 24, 2026
@pingcap-cla-assistant

pingcap-cla-assistant Bot commented Jul 24, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

0xTars added 2 commits July 24, 2026 14:44
Pin the coordinated kvproto and client-go revisions together so every
downstream commit has both the explicit merged-task acknowledgement and
nested child-accounting support.

Use the replacement PR forks until their upstream changes are available
from stable module revisions. Keeping both pins in one commit avoids
transient dependency states that expose only half of the cross-repo
contract.

Signed-off-by: 0xTars <1196089730@qq.com>
Capture the existing boundaries before enabling the feature: Analyze
requests do not join store batching yet, and tasks without a per-task
response remain eligible for retry.

Keep the new unanswered-response scenario and existing bucket-version
scenario as sequential helper calls under one top-level test. This
preserves the package limit of 50 tests without adding subtests to a
sequential regression suite.

Signed-off-by: 0xTars <1196089730@qq.com>
Enable store batching only for V2 full-sampling Analyze requests. Derive
the outer RPC concurrency and child count from the resolved Analyze scan
budget in the same commit, so batching cannot multiply either manual or
system Analyze concurrency.

Carry an explicit merge-capability marker from AnalyzeColumnsExec
through task construction and the outgoing protobuf request. Keep other
Analyze callers on the non-batched path, and cover the opt-in boundary
and actual wire negotiation.

Allow TiKV to merge acknowledged child results into the main response.
Consume those acknowledgements without emitting empty responses, retain
their execution details for runtime and runaway accounting, and keep
unanswered tasks on the existing retry path.

Use a maximum physical width of five scans, leave KeepOrder disabled for
the sample-based consumer, and bypass the row-count-hint gate only after
explicit opt-in. Extend the behavior tests to cover bounded batching and
merged acknowledgements.

Signed-off-by: 0xTars <1196089730@qq.com>
@ti-chi-bot

ti-chi-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign 0xpoe, cfzjywxk for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution This PR is from a community contributor. do-not-merge/needs-triage-completed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant