planner: flaky test TestBatchDropBindings

[terry1purcell]

What problem does this PR solve?

Issue Number: close #66371

Problem Summary:

What changed and how does it work?

Decision was made here to fix the test rather than the underlying behavior. The test does expose a race condition - but the likelihood that a customer sees this as a critical issue, or that it has any significant impact to their operations - is arguably low.

The issue is - if the binding cache reload (which occurs every 3 seconds) had begun before the drop completed, then the dropped binding could be reloaded - and exist for 3 seconds longer than the user intended. The solution for that would be to add a mutex to the bindingCacheUpdater - but that would then be executed every 3 seconds. However, if a customer exposes this problem - then it may be necessary to add this fix.

Analysis of the issue:

Root Cause: Race Between Background Binding Loader and DROP

The test does not set bindinfo.Lease = 0, so the background goroutine globalBindHandleWorkerLoop (started in domain.go:1571) runs every 3 seconds, calling LoadFromStorageToCache(false, false). This races with the DropBinding operation's own deferred LoadFromStorageToCache call.

The Race in Detail

LoadFromStorageToCache (binding_cache.go:64) is not atomic — it first reads bindings from storage via SQL, then iterates over them updating the cache one-by-one. There is no mutex protecting the entire read-process cycle. Two concurrent calls can interleave.

Here's the problematic sequence:

┌──────┬─────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────┐
  │ Step │                    Background goroutine                     │                         Test goroutine                         │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │ 1    │ LoadFromStorageToCache starts, reads storage snapshot →     │                                                                │
  │      │ sees enabled bindings (update_time=T0)                      │                                                                │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │ 2    │                                                             │ DROP binding commits → marks bindings as deleted               │
  │      │                                                             │ (update_time=T1 > T0) in mysql.bind_info                       │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │      │                                                             │ DROP's deferred LoadFromStorageToCache reads storage → sees    │
  │ 3    │                                                             │ deleted bindings → pickCachedBinding(enabled@T0, deleted@T1) → │
  │      │                                                             │  nil → RemoveBinding() → cache is now empty                    │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │      │ Continues processing stale snapshot: GetBinding() returns   │                                                                │
  │ 4    │ nil (just removed). pickCachedBinding(nil, enabled@T0) →    │                                                                │
  │      │ returns enabled@T0 → SetBinding() re-adds the binding to    │                                                                │
  │      │ cache                                                       │                                                                │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │ 5    │                                                             │ SHOW BINDINGS → sees the re-added binding → assertion fails    │
  └──────┴─────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────┘

Why the stale data persists

At step 4, the background goroutine also stores lastUpdateTime = T0 (from its stale snapshot), potentially overwriting the DROP's stored T1. This makes lastUpdateTime go backwards. The 10-second timeLagTolerance (binding_cache.go:94) means the next incremental load will likely catch the T1 deleted record again, but there's a transient window (up to the next 3-second tick) where the cache has stale data.

Key code locations

• Background loop: pkg/domain/domain.go:1571-1626 — ticks every bindinfo.Lease (default 3s)
• Lease default: pkg/bindinfo/binding_handle.go:26 — var Lease = 3 * time.Second
• Lease=0 check: pkg/domain/domain.go:1560 — if bindinfo.Lease == 0 { return } (skips starting the background goroutine)
• DropBinding deferred load: pkg/bindinfo/binding_operator.go:164-167
• Non-atomic LoadFromStorageToCache: pkg/bindinfo/binding_cache.go:64-136 — reads from storage, then processes into cache without a mutex
• Mock owner succeeds: pkg/owner/mock.go:135 — CampaignOwner() returns nil, so the background loop does start in mock test environments

Evidence: other tests avoid this

TestGCBindRecord (bind_test.go:371-377) explicitly sets bindinfo.Lease = 0 before creating the mock store/domain, preventing the background goroutine from starting. TestBatchDropBindings does not.

Session bindings are not affected

removeAllBindings(tk, false) for session bindings uses DropSessionBinding (session_handle.go:90), which is a simple in-memory map delete with no background reload — no race possible there.

Summary

The flakiness is caused by the background globalBindHandleWorkerLoop goroutine's LoadFromStorageToCache interleaving with the DropBinding deferred LoadFromStorageToCache, re-adding stale (enabled) bindings to the cache after the DROP already removed them. The fix would be to set bindinfo.Lease = 0 at the start of the test (as TestGCBindRecord does), or to add mutual exclusion around LoadFromStorageToCache.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[pantheon-ai]

Review Failed

Environment preparation failed after 3 attempts due to infrastructure service unavailability (503 Service Unavailable).

Please retry the review request later when the upstream services recover.

[tiprow]

Hi @terry1purcell. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[terry1purcell]

/ok-to-test

[Copilot]

Pull request overview

Fixes flakiness in TestBatchDropBindings by making the bindinfo lease deterministic during the test, preventing background binding worker activity from introducing timing-dependent behavior.

Changes:

• Temporarily set bindinfo.Lease to 0 at the start of TestBatchDropBindings.
• Restore the original bindinfo.Lease value via defer to avoid leaking global state across tests.

[terry1purcell]

@pantheon-bot please review

[pantheon-ai]

Review Failed

Environment preparation failed after 3 attempts due to infrastructure service unavailability (503 Service Unavailable).

Please retry the review request later when the upstream services recover.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.5654%. Comparing base (4895d30) to head (0fe8f2d).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #66559        +/-   ##
================================================
- Coverage   77.6882%   77.5654%   -0.1229%     
================================================
  Files          2006       1928        -78     
  Lines        548796     536467     -12329     
================================================
- Hits         426350     416113     -10237     
+ Misses       120785     120348       -437     
+ Partials       1661          6      -1655     

Flag	Coverage Δ
integration	41.6781% <ø> (-6.5041%)	⬇️
unit	76.6933% <ø> (+0.3479%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	56.7974% <ø> (ø)
parser	∅ <ø> (∅)
br	48.8076% <ø> (-12.0755%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[terry1purcell]

/retest-required

[terry1purcell]

/retest

[terry1purcell]

/retest

[mjonss]

Is this a test issue and not an actual issue with DROP BINDING?

[mjonss]

LGTM

[fixdb]

+1

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fixdb, mjonss

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/bindinfo/OWNERS [fixdb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-02-26 21:49:19.746155636 +0000 UTC m=+391632.260950255: ☑️ agreed by mjonss.
• 2026-02-26 23:27:53.690625308 +0000 UTC m=+397546.205419937: ☑️ agreed by fixdb.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.5: #70013.