From 5c24b3d5050b23b57c187bfb4dbec564e71308de Mon Sep 17 00:00:00 2001 From: Abhishek Pal Date: Fri, 10 Jul 2026 17:18:01 +0530 Subject: [PATCH 1/2] RATIS-2599. Add design for listener delegation from followers --- .../markdown/designs/listener-delegation.md | 183 ++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 ratis-docs/src/site/markdown/designs/listener-delegation.md diff --git a/ratis-docs/src/site/markdown/designs/listener-delegation.md b/ratis-docs/src/site/markdown/designs/listener-delegation.md new file mode 100644 index 0000000000..c23b08cdf6 --- /dev/null +++ b/ratis-docs/src/site/markdown/designs/listener-delegation.md @@ -0,0 +1,183 @@ +--- +title: Listener Replication Delegation +summary: Delegate log replication to listeners from the leader to sufficiently caught-up followers +date: 2026-07-10 +jira: RATIS-2599 +status: proposed +author: Abhishek Pal +--- + + + +# Listener Replication Delegation (AppendEntries from Followers) + +## Summary + +This document describes the design for delegating log replication to listeners from the leader to sufficiently caught-up followers. +Instead of the leader sending AppendEntries RPCs directly to every listener, a caught-up follower can serve as the replication source for one or more listeners. This reduces leader load in clusters with many listeners and can reduce network hops for co-located listener-follower pairs. + +## Motivation + +In the current architecture, the leader maintains a `LogAppender` for every peer in the cluster — both followers (voting members) and listeners (non-voting members). +This means: + +1. **Leader bandwidth bottleneck:** In clusters with many listeners, the leader must send every log entry N times — once per listener. This concentrates network I/O on a single node. + +2. **Unnecessary network hops:** Listeners co-located with a follower (same rack/region) still receive data from a potentially remote leader. + +3. **Leader CPU overhead:** Serializing and managing per-listener LogAppender state consumes leader resources that could be used for consensus-critical work. + +The proposed solution offloads listener replication to followers that already have the data, freeing the leader to focus on consensus with voting members. + +## Design Overview + +``` +BEFORE (current): AFTER (proposed): +┌────────┐ ┌────────┐ +│ Leader │──appendEntries──► Follower1 │ Leader │──appendEntries──► Follower1 +│ │──appendEntries──► Follower2 │ │──appendEntries──► Follower2 +│ │──appendEntries──► Listener1 │ │──heartbeat-only──► Listener1 +│ │──appendEntries──► Listener2 │ │──heartbeat-only──► Listener2 +└────────┘ └────────┘ + │ (delegation notification) + ▼ + Follower1──appendEntries──► Listener1 + Follower2──appendEntries──► Listener2 +``` + +### Key Invariants + +1. **Leader authority:** The leader makes all delegation decisions. Followers do not self-elect to serve listeners. + +2. **Leader heartbeat continuity:** The leader continues sending full heartbeats (empty entries with `leaderCommit` and `commitInfos`) directly to delegated listeners. This maintains liveness detection and ensures listeners get accurate commit progress from the authoritative source. + +3. **Term safety:** Listeners validate that `leaderTerm >= currentTerm` in delegated entries. A deposed leader's follower will have a stale term, so listeners reject stale entries automatically. + +4. **Log consistency:** The standard Raft log consistency check (`previousLog` matching) still runs on the listener side preventing any divergence regardless of the source. + +5. **Full catch-up support:** The follower replicator can serve listeners from any `nextIndex` using its local log — not limited to near-real-time forwarding. + +6. **Automatic fallback:** If the delegate follower becomes unhealthy, the leader revokes delegation and resumes direct replication. The listener is unaware of the switch. + +## Detailed Design + +### 1. Protocol Changes + +Extend the existing `AppendEntriesRequest/Reply` messages with new fields. + +```protobuf +// Leader → Follower: "here are the listeners you should serve" +message ListenerAssignmentProto { + repeated RaftPeerProto assignedListeners = 1; + uint64 leaderTerm = 2; +} + +// Follower → Leader: "here's how each listener is progressing" +message ListenerProgressProto { + RaftPeerIdProto listenerId = 1; + uint64 matchIndex = 2; + uint64 nextIndex = 3; + uint64 commitIndex = 4; +} + +// Extended AppendEntriesRequestProto: +// field 16: ListenerAssignmentProto listenerAssignment +// field 17: bool fromFollower + +// Extended AppendEntriesReplyProto: +// field 8: repeated ListenerProgressProto listenerProgress +``` + +### 2. Configuration + +All configuration under `raft.server.listener.delegation.*`: + +| Key | Default | Description | +|-----|---------|-------------| +| `enabled` | `false` | Master switch for the feature | +| `max-listeners-per-follower` | `5` | Maximum listeners a single follower can serve | +| `follower-lag-threshold` | `100` | Max entries a follower can lag behind leader and still be eligible | +| `reassignment-interval` | `30s` | How often the leader re-evaluates assignments | + +### 3. Leader-side: Selection and Assignment + +**ListenerDelegationSelector** (stateless utility, mirrors `SnapshotSourceSelector`): +- Filters followers: `leaderLastIndex - follower.matchIndex <= lagThreshold` +- Ranks by: highest matchIndex, then freshest RPC response +- Load-balances: round-robin assignment respecting `maxListenersPerFollower` +- Returns `Map>` — empty map triggers leader-direct fallback + +**ListenerDelegationState** (mutable, held by `LeaderStateImpl`): +- Tracks `listener → delegateFollower` mapping +- Tracks `follower → [assignedListeners]` mapping +- Marks followers with pending notification changes +- Provides `revokeDelegation(listenerId)` and `clear()` (term change) + +**Leader LogAppender behavior for delegated listeners:** +- Switches to **heartbeat-only mode**: sends periodic heartbeats with accurate + `leaderCommit` and `commitInfos`, but no log entries +- Immediately resumes full mode on revocation + +### 4. Follower-side: Replication to Listeners + +**FollowerListenerReplicator** (daemon thread per assigned listener): +- Reads entries from the follower's local `RaftLog` +- Builds `AppendEntriesRequestProto` with `fromFollower=true` and the leader's term +- Tracks `nextIndex` / `matchIndex` per listener +- Handles `INCONSISTENCY` replies by adjusting `nextIndex` (full catch-up support) +- Uses the follower's `commitIndex` as `leaderCommit` (safe lower bound) +- Stops immediately on: term change, assignment revocation, `NOT_LEADER` reply +- If listener needs entries the follower no longer has (purged by snapshot): + reports failure to leader for snapshot-based recovery via RATIS-2428 + +**FollowerListenerReplicatorManager** (lifecycle manager): +- Processes `listenerAssignment` from leader's AppendEntries +- Starts/stops/updates replicators based on assignment changes +- Collects `ListenerProgressProto` reports for piggybacking on follower's reply + +### 5. Listener-side: Accepting Delegated Entries + +Modified `RaftServerImpl.appendEntriesAsync()`: + +- **When `fromFollower=true` AND node is LISTENER:** + - Validate `leaderTerm >= currentTerm` (reject stale terms) + - Do NOT call `state.setLeader(senderId)` — listener's known leader stays unchanged + - Do NOT call `changeToFollowerAndPersistMetadata` — listener stays a listener + - DO update `lastRpcTime` (prevents timeout alerts) + - DO proceed with standard log append and consistency checks +- **When `fromFollower=true` AND node is NOT LISTENER:** reject with `NOT_LEADER` +- **When `fromFollower=false`:** existing logic unchanged + +### 6. Leader-side: Progress Monitoring and Fallback + +**Progress processing:** +- When handling `AppendEntriesReply` from a delegate follower, extract + `listenerProgress` reports +- Update leader's own `FollowerInfo` for each reported listener (matchIndex, + commitIndex) + +**Health checks (every `reassignment-interval`):** +- Delegate follower's matchIndex fell behind `lagThreshold` → revoke +- Listener's matchIndex not advancing → revoke +- Delegate follower unresponsive → revoke +- Cooldown period prevents rapid reassignment oscillation + +**Revocation:** +- Resume leader's LogAppender for that listener (heartbeat-only → full) +- Notify follower with empty `assignedListeners` on next heartbeat +- Follower's `FollowerListenerReplicatorManager` stops the replicator From b93e54b7d819cdefd0f5ccbfe0ea099b30758a81 Mon Sep 17 00:00:00 2001 From: Abhishek Pal Date: Sat, 25 Jul 2026 21:45:46 +0530 Subject: [PATCH 2/2] Address review comments --- .../markdown/designs/listener-delegation.md | 226 +++++++++++------- 1 file changed, 138 insertions(+), 88 deletions(-) diff --git a/ratis-docs/src/site/markdown/designs/listener-delegation.md b/ratis-docs/src/site/markdown/designs/listener-delegation.md index c23b08cdf6..58525d8aac 100644 --- a/ratis-docs/src/site/markdown/designs/listener-delegation.md +++ b/ratis-docs/src/site/markdown/designs/listener-delegation.md @@ -1,6 +1,6 @@ --- title: Listener Replication Delegation -summary: Delegate log replication to listeners from the leader to sufficiently caught-up followers +summary: Delegate log replication to listeners from the leader to admin-selected followers date: 2026-07-10 jira: RATIS-2599 status: proposed @@ -28,21 +28,36 @@ author: Abhishek Pal ## Summary -This document describes the design for delegating log replication to listeners from the leader to sufficiently caught-up followers. -Instead of the leader sending AppendEntries RPCs directly to every listener, a caught-up follower can serve as the replication source for one or more listeners. This reduces leader load in clusters with many listeners and can reduce network hops for co-located listener-follower pairs. +This document describes the design for delegating log replication to listeners +from the leader to a follower. +Instead of the leader sending AppendEntries RPCs directly to every listener, +a follower can serve as the replication source for one or more listeners. +This reduces leader load in clusters with many listeners +and can reduce network hops for co-located listener-follower pairs. ## Motivation -In the current architecture, the leader maintains a `LogAppender` for every peer in the cluster — both followers (voting members) and listeners (non-voting members). +In the current architecture, +the leader maintains a `LogAppender` for every peer in the cluster — +both followers (voting members) and listeners (non-voting members). This means: -1. **Leader bandwidth bottleneck:** In clusters with many listeners, the leader must send every log entry N times — once per listener. This concentrates network I/O on a single node. +1. **Leader bandwidth bottleneck:** + In clusters with many listeners, + the leader must send every log entry N times — once per listener. + This concentrates network I/O on a single node. -2. **Unnecessary network hops:** Listeners co-located with a follower (same rack/region) still receive data from a potentially remote leader. +2. **Unnecessary network hops:** + Listeners co-located with a follower (same rack/region) + still receive data from a potentially remote leader. -3. **Leader CPU overhead:** Serializing and managing per-listener LogAppender state consumes leader resources that could be used for consensus-critical work. +3. **Leader CPU overhead:** + Serializing and managing per-listener `LogAppender` state + consumes leader resources that could be used for consensus-critical work. -The proposed solution offloads listener replication to followers that already have the data, freeing the leader to focus on consensus with voting members. +The proposed solution offloads listener replication to a follower that already +has the data, +freeing the leader to focus on consensus with voting members. ## Design Overview @@ -62,23 +77,47 @@ BEFORE (current): AFTER (proposed): ### Key Invariants -1. **Leader authority:** The leader makes all delegation decisions. Followers do not self-elect to serve listeners. - -2. **Leader heartbeat continuity:** The leader continues sending full heartbeats (empty entries with `leaderCommit` and `commitInfos`) directly to delegated listeners. This maintains liveness detection and ensures listeners get accurate commit progress from the authoritative source. - -3. **Term safety:** Listeners validate that `leaderTerm >= currentTerm` in delegated entries. A deposed leader's follower will have a stale term, so listeners reject stale entries automatically. - -4. **Log consistency:** The standard Raft log consistency check (`previousLog` matching) still runs on the listener side preventing any divergence regardless of the source. - -5. **Full catch-up support:** The follower replicator can serve listeners from any `nextIndex` using its local log — not limited to near-real-time forwarding. - -6. **Automatic fallback:** If the delegate follower becomes unhealthy, the leader revokes delegation and resumes direct replication. The listener is unaware of the switch. +1. **Leader/admin authority:** + The delegation topology is decided by the administrator + and enforced by the leader. + Followers do not self-elect to serve listeners. + *Future work:* a listener may select which follower to append from, + but for now the admin/leader decides. + +2. **Leader heartbeat continuity:** + The leader continues sending full heartbeats + (empty entries with `leaderCommit` and `commitInfos`) + directly to delegated listeners. + This maintains liveness detection + and ensures listeners get accurate commit progress from the authoritative source. + +3. **Term safety:** + Listeners validate that `leaderTerm >= currentTerm` in delegated entries. + A deposed leader's follower will have a stale term, + so listeners reject stale entries automatically. + +4. **Log consistency (same as before):** + The standard Raft log consistency check (`previousLog` matching) + still runs on the listener side, preventing any divergence regardless of the source. + This is the exact same check the listener applies to entries from the leader today — + only the sending peer differs. + +5. **Full catch-up support (same as before):** + The follower serves the listener using the same append/catch-up logic + the leader uses to catch up any peer today. + The follower can serve the listener from any `nextIndex` using its local log — + there is no new catch-up mechanism, only a different source. + +6. **Automatic fallback:** + If the delegate follower becomes unhealthy, + the leader resumes direct replication. + The listener is unaware of the switch. ## Detailed Design ### 1. Protocol Changes -Extend the existing `AppendEntriesRequest/Reply` messages with new fields. +Extend the existing `AppendEntriesRequest` message with new fields. ```protobuf // Leader → Follower: "here are the listeners you should serve" @@ -87,97 +126,108 @@ message ListenerAssignmentProto { uint64 leaderTerm = 2; } -// Follower → Leader: "here's how each listener is progressing" -message ListenerProgressProto { - RaftPeerIdProto listenerId = 1; - uint64 matchIndex = 2; - uint64 nextIndex = 3; - uint64 commitIndex = 4; -} - // Extended AppendEntriesRequestProto: // field 16: ListenerAssignmentProto listenerAssignment // field 17: bool fromFollower - -// Extended AppendEntriesReplyProto: -// field 8: repeated ListenerProgressProto listenerProgress ``` -### 2. Configuration +No progress message is added on the reply path. +The leader already heartbeats listeners directly (invariant 2), +so each listener's own `AppendEntriesReply` to the leader +already carries its `matchIndex` / `commitIndex`. +The leader therefore learns listener progress directly from the listeners — +no follower → leader progress channel is needed. -All configuration under `raft.server.listener.delegation.*`: +### 2. Configuration | Key | Default | Description | |-----|---------|-------------| -| `enabled` | `false` | Master switch for the feature | -| `max-listeners-per-follower` | `5` | Maximum listeners a single follower can serve | -| `follower-lag-threshold` | `100` | Max entries a follower can lag behind leader and still be eligible | -| `reassignment-interval` | `30s` | How often the leader re-evaluates assignments | - -### 3. Leader-side: Selection and Assignment - -**ListenerDelegationSelector** (stateless utility, mirrors `SnapshotSourceSelector`): -- Filters followers: `leaderLastIndex - follower.matchIndex <= lagThreshold` -- Ranks by: highest matchIndex, then freshest RPC response -- Load-balances: round-robin assignment respecting `maxListenersPerFollower` -- Returns `Map>` — empty map triggers leader-direct fallback +| `raft.server.listener.delegation.enabled` | `false` | Master switch for the feature | + +The delegation topology itself is **not** auto-tuned by config knobs. +It is supplied by the administrator (see §3), +who knows the cluster topology (racks/regions) far better than an automatic selector. + +### 3. Admin-configured Assignment + +Rather than have the leader automatically rank and pick source followers, +the administrator declares the listener → source-follower mapping. +This mirrors how peers and listeners are already managed today +through `setConfiguration` +(`AdminApi.setConfiguration(servers, listeners)`, +carried in `SetConfigurationRequestProto.listeners`). + +`RaftPeerProto` currently has no free-form metadata field +(it carries `id`, `address`, `priority`, `dataStreamAddress`, +`clientAddress`, `adminAddress`, `startupRole`). +To express the mapping, +add an optional source-follower field on the listener peer, +e.g. `optional bytes delegateSource` on `RaftPeerProto`, +set by the admin when submitting the listener list. +It flows through the existing pipeline: +`SetConfigurationRequest` → `PeerConfiguration` → Raft log → `LeaderStateImpl`. + +**Leader behavior:** +- Reads the admin-supplied listener → follower mapping from the configuration. +- Sends `listenerAssignment` to the chosen follower + (piggybacked on the follower's AppendEntries). +- Switches each delegated listener's `LogAppender` to **heartbeat-only mode**: + periodic heartbeats with accurate `leaderCommit` and `commitInfos`, but no log entries. +- Immediately resumes full mode on fallback (see §6). **ListenerDelegationState** (mutable, held by `LeaderStateImpl`): -- Tracks `listener → delegateFollower` mapping -- Tracks `follower → [assignedListeners]` mapping -- Marks followers with pending notification changes -- Provides `revokeDelegation(listenerId)` and `clear()` (term change) - -**Leader LogAppender behavior for delegated listeners:** -- Switches to **heartbeat-only mode**: sends periodic heartbeats with accurate - `leaderCommit` and `commitInfos`, but no log entries -- Immediately resumes full mode on revocation +- Tracks `listener → delegateFollower` mapping (from configuration). +- Tracks `follower → [assignedListeners]` mapping. +- Marks followers with pending notification changes. +- Provides `clear()` on term change. ### 4. Follower-side: Replication to Listeners **FollowerListenerReplicator** (daemon thread per assigned listener): -- Reads entries from the follower's local `RaftLog` -- Builds `AppendEntriesRequestProto` with `fromFollower=true` and the leader's term -- Tracks `nextIndex` / `matchIndex` per listener -- Handles `INCONSISTENCY` replies by adjusting `nextIndex` (full catch-up support) -- Uses the follower's `commitIndex` as `leaderCommit` (safe lower bound) -- Stops immediately on: term change, assignment revocation, `NOT_LEADER` reply -- If listener needs entries the follower no longer has (purged by snapshot): - reports failure to leader for snapshot-based recovery via RATIS-2428 +- Reads entries from the follower's local `RaftLog`. +- Builds `AppendEntriesRequestProto` with `fromFollower=true` and the leader's term. +- Tracks `nextIndex` / `matchIndex` per listener. +- Handles `INCONSISTENCY` replies by adjusting `nextIndex` (full catch-up support). +- Uses the follower's `commitIndex` as `leaderCommit` (safe lower bound). +- Stops immediately on: term change, assignment revocation, `NOT_LEADER` reply. +- If the listener needs entries the follower no longer has (purged by snapshot): + reports failure to the leader for snapshot-based recovery via RATIS-2428. **FollowerListenerReplicatorManager** (lifecycle manager): -- Processes `listenerAssignment` from leader's AppendEntries -- Starts/stops/updates replicators based on assignment changes -- Collects `ListenerProgressProto` reports for piggybacking on follower's reply +- Processes `listenerAssignment` from the leader's AppendEntries. +- Starts/stops/updates replicators based on assignment changes. ### 5. Listener-side: Accepting Delegated Entries Modified `RaftServerImpl.appendEntriesAsync()`: - **When `fromFollower=true` AND node is LISTENER:** - - Validate `leaderTerm >= currentTerm` (reject stale terms) - - Do NOT call `state.setLeader(senderId)` — listener's known leader stays unchanged - - Do NOT call `changeToFollowerAndPersistMetadata` — listener stays a listener - - DO update `lastRpcTime` (prevents timeout alerts) - - DO proceed with standard log append and consistency checks -- **When `fromFollower=true` AND node is NOT LISTENER:** reject with `NOT_LEADER` -- **When `fromFollower=false`:** existing logic unchanged + - Validate `leaderTerm >= currentTerm` (reject stale terms). + - Do NOT call `state.setLeader(senderId)` — the listener's known leader stays unchanged. + - Do NOT call `changeToFollowerAndPersistMetadata` — the listener stays a listener. + - DO update `lastRpcTime` (prevents timeout alerts). + - DO proceed with standard log append and consistency checks. +- **When `fromFollower=true` AND node is NOT LISTENER:** reject with `NOT_LEADER`. +- **When `fromFollower=false`:** existing logic unchanged. ### 6. Leader-side: Progress Monitoring and Fallback -**Progress processing:** -- When handling `AppendEntriesReply` from a delegate follower, extract - `listenerProgress` reports -- Update leader's own `FollowerInfo` for each reported listener (matchIndex, - commitIndex) - -**Health checks (every `reassignment-interval`):** -- Delegate follower's matchIndex fell behind `lagThreshold` → revoke -- Listener's matchIndex not advancing → revoke -- Delegate follower unresponsive → revoke -- Cooldown period prevents rapid reassignment oscillation - -**Revocation:** -- Resume leader's LogAppender for that listener (heartbeat-only → full) -- Notify follower with empty `assignedListeners` on next heartbeat -- Follower's `FollowerListenerReplicatorManager` stops the replicator +**Progress:** +- The leader reads each listener's `matchIndex` / `commitIndex` + from the listener's own direct-heartbeat replies — + the same replies it already receives today. +- No progress is extracted from the delegate follower. + +**Health checks and fallback:** +- If the delegate follower becomes unresponsive + (detected from its own AppendEntries replies), + the leader resumes direct replication for the affected listeners + (heartbeat-only → full mode). +- On recovery, the leader resumes the admin-configured delegation. +- The leader does not auto-reassign a listener to a different follower; + the admin owns the topology. + +**Revocation (admin change or term change):** +- Resume the leader's `LogAppender` for that listener (heartbeat-only → full). +- Notify the follower with an empty `assignedListeners` on the next heartbeat. +- The follower's `FollowerListenerReplicatorManager` stops the replicator.