fix(balancer): keep least_conn in-flight count across upstream scaling#13666
Open
AlinsRan wants to merge 2 commits into
Open
fix(balancer): keep least_conn in-flight count across upstream scaling#13666AlinsRan wants to merge 2 commits into
AlinsRan wants to merge 2 commits into
Conversation
The least_conn picker stores per-server connection scores inside a binary heap that lives in the picker instance. The picker is cached by upstream version and rebuilt whenever the upstream changes, e.g. when nodes are scaled. On rebuild every score is reset to the base weight, so connections already established are forgotten. For long-lived connections such as WebSocket this leaves the load skewed on the original nodes and the newly added nodes are not preferred, degrading least_conn to round-robin. Keep a per-worker in-flight connection count for each (upstream, server) outside the picker, keyed by the upstream resource key which is stable across scaling. The rebuilt heap seeds each score from this count, so surviving nodes keep their load while freshly added nodes start empty and are preferred right after scaling. Drained servers that leave the upstream are pruned to bound memory. Fixes #12217
Clamp the persisted per-server connection count at zero. On retry exhaustion balancer.lua releases the last picked server once on retry entry and again in the log phase; without the clamp that double release drove the count negative and, unlike the old per-version heap, the negative value survived balancer recreation and over-preferred the server on the next rebuild. Also add a drain test (a fully released node returns to the pool at baseline after recreation) to cover the decrement path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When
least_connproxies long-lived connections (e.g. WebSocket) and the upstream is scaled, load stays skewed on the original nodes and the newly added nodes are not preferred —least_conneffectively degrades to round-robin.Root cause. The picker keeps per-server scores in a binary heap that lives inside the picker instance. The picker is cached by upstream version and rebuilt whenever the upstream changes (scaling bumps the version). On rebuild every score is reset to the base weight
1/weight, so the connections already established on the old nodes are forgotten. New connections are then spread evenly across all nodes instead of favoring the empty new node, so the imbalance persists for the whole lifetime of the long connections.Fix. Keep a per-worker in-flight connection count for each
(upstream, server)outside the picker, keyed by the upstreamresource_key/resource_idwhich is stable across scaling (unlike the picker version). The rebuilt heap seeds each score with(count + 1) / weight, so surviving nodes keep their load while freshly added nodes start empty and are preferred right after scaling. The count is incremented/decremented in exactly the same places the heap score already moves (get/after_balance), so it always equals the current in-flight count. Servers that leave the upstream and have drained are pruned to bound memory on node churn.No new config, no shared dict, no opt-in flag: this is the intended
least_connbehavior, so it is on by default. Counting is per-worker, matching the existing per-worker heap — each worker balances its own connections. For short requests the count returns to zero as soon as each request completes, so steady-state behavior is unchanged.Which issue(s) this PR fixes:
Fixes #12217
Notes for reviewers
This is an alternative to #12261 for the same issue. Differences:
lua_shared_dictplumbing; a worker only ever balances its own connections, so a per-worker count is both simpler and more accurate than a global one.resource_key/resource_id. fix: WebSocket load balancing imbalance with least_conn after upstream scaling #12261's fallback hashes the whole upstream config (which includesnodes), so the key changes on the very scaling event it targets and the persisted count is lost for id-less upstreams.persistent_conn_countingflag. The imbalance is surprising default behavior, so the fix should apply without users having to discover a switch.least_connusers are unaffected.Checklist