Skip to content

fix(balancer): keep least_conn in-flight count across upstream scaling#13666

Open
AlinsRan wants to merge 2 commits into
masterfrom
fix/least-conn-conn-count-persist
Open

fix(balancer): keep least_conn in-flight count across upstream scaling#13666
AlinsRan wants to merge 2 commits into
masterfrom
fix/least-conn-conn-count-persist

Conversation

@AlinsRan

@AlinsRan AlinsRan commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Description

When least_conn proxies long-lived connections (e.g. WebSocket) and the upstream is scaled, load stays skewed on the original nodes and the newly added nodes are not preferred — least_conn effectively degrades to round-robin.

Root cause. The picker keeps per-server scores in a binary heap that lives inside the picker instance. The picker is cached by upstream version and rebuilt whenever the upstream changes (scaling bumps the version). On rebuild every score is reset to the base weight 1/weight, so the connections already established on the old nodes are forgotten. New connections are then spread evenly across all nodes instead of favoring the empty new node, so the imbalance persists for the whole lifetime of the long connections.

Fix. Keep a per-worker in-flight connection count for each (upstream, server) outside the picker, keyed by the upstream resource_key/resource_id which is stable across scaling (unlike the picker version). The rebuilt heap seeds each score with (count + 1) / weight, so surviving nodes keep their load while freshly added nodes start empty and are preferred right after scaling. The count is incremented/decremented in exactly the same places the heap score already moves (get / after_balance), so it always equals the current in-flight count. Servers that leave the upstream and have drained are pruned to bound memory on node churn.

No new config, no shared dict, no opt-in flag: this is the intended least_conn behavior, so it is on by default. Counting is per-worker, matching the existing per-worker heap — each worker balances its own connections. For short requests the count returns to zero as soon as each request completes, so steady-state behavior is unchanged.

Which issue(s) this PR fixes:

Fixes #12217

Notes for reviewers

This is an alternative to #12261 for the same issue. Differences:

  • Per-worker table instead of a shared dict. Avoids cross-worker races and new nginx lua_shared_dict plumbing; a worker only ever balances its own connections, so a per-worker count is both simpler and more accurate than a global one.
  • Stable key. Keyed on resource_key/resource_id. fix: WebSocket load balancing imbalance with least_conn after upstream scaling #12261's fallback hashes the whole upstream config (which includes nodes), so the key changes on the very scaling event it targets and the persisted count is lost for id-less upstreams.
  • On by default, no persistent_conn_counting flag. The imbalance is surprising default behavior, so the fix should apply without users having to discover a switch.
  • Single-node fast path kept, so regular least_conn users are unaffected.
  • Memory bounded via pruning of drained, removed servers.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible

The least_conn picker stores per-server connection scores inside a binary
heap that lives in the picker instance. The picker is cached by upstream
version and rebuilt whenever the upstream changes, e.g. when nodes are
scaled. On rebuild every score is reset to the base weight, so connections
already established are forgotten. For long-lived connections such as
WebSocket this leaves the load skewed on the original nodes and the newly
added nodes are not preferred, degrading least_conn to round-robin.

Keep a per-worker in-flight connection count for each (upstream, server)
outside the picker, keyed by the upstream resource key which is stable
across scaling. The rebuilt heap seeds each score from this count, so
surviving nodes keep their load while freshly added nodes start empty and
are preferred right after scaling. Drained servers that leave the upstream
are pruned to bound memory.

Fixes #12217
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Jul 7, 2026
Clamp the persisted per-server connection count at zero. On retry
exhaustion balancer.lua releases the last picked server once on retry
entry and again in the log phase; without the clamp that double release
drove the count negative and, unlike the old per-version heap, the
negative value survived balancer recreation and over-preferred the
server on the next rebuild.

Also add a drain test (a fully released node returns to the pool at
baseline after recreation) to cover the decrement path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

help request: WebSocket Load Balancing Imbalance Issue After Upstream Node Scaling

1 participant