Skip to content

lifecycle: adapt management producers and synchronous APIs - #1269

Open
dkropachev wants to merge 5 commits into
lifecycle/1257-reactor-completionfrom
lifecycle/1257-runtime-adapters
Open

dkropachev wants to merge 5 commits into
lifecycle/1257-reactor-completionfrom
lifecycle/1257-runtime-adapters

Conversation

@dkropachev

@dkropachev dkropachev commented Nov 14, 2026 •

Copy link
Copy Markdown

Move existing lifecycle notification delivery onto the bounded management runtime while preserving the publication locks and pool recovery ordering. Bind the runtime lazily to the selected reactor and detach its channel after the final cleanup barrier.

DOWN/REMOVE producers retain coalesced negative observations when ordinary I/O or callback credits are full. Deferred positive observations follow previously admitted events and retain causal family ordering. Unknown-host overload is explicit. Transaction readiness is handled by the scheduler; delivery handlers do not wait for the transaction caller. Nested asynchronous recovery joins the active drainer, and failed executor handoffs restore idle state.

Synchronous connect, refresh and keyspace APIs wait for admission before side effects. Callback reentry uses a separate aggregate and bounded prerequisite window. Policies run between released compatibility phases. Blocking APIs reject actual reactor and owner threads. Management-context shutdown defers legacy waits to the existing executor and retains cleanup through the final barrier, including late results. First-use startup cannot install a runtime after shutdown, and failed pool admission releases host-UP handling so recovery can retry.

Fixes #1257. Final part of the stack, dependent on #1268 and #1267, with #1255 beneath them. The ownership contract proposed in #1265 and the prerequisite PRs remain under review. Actual cluster/pool field migration remains in later issues. Ordinary query routing gains no mailbox hop and protocol behavior is unchanged. Workload and performance-budget agreement and benchmark measurements remain outstanding.

Validation on the final source tree: 1,102 tests and 56 subtests pass with rebuilt Cython modules, and the same 1,102 tests and 56 subtests pass with pure-Python core modules. Both configurations also pass all 42 asyncio reactor tests. The broad suites include real libev and Python 3.11 asyncore coverage. PyPy passes 1,054 tests and 56 subtests plus the 42 asyncio tests; 18 native-libev cases are explicitly deselected because that extension is unavailable on PyPy here. Controlled regressions cover saturation, synchronous callback reentry, 128-pool fanout with one executor worker, raw reactor API rejection, managed shutdown, startup unwind, nested recovery and collection. TZ=UTC is used consistently and git diff --check passes. Live-cluster integration and performance measurements remain outstanding.

Pre-review checklist

  • I have split my patch into logically separate commits.
  • All commit messages clearly explain what they change and why.
  • I added relevant tests for new features and bug fixes.
  • All commits compile, pass static checks and pass test.
  • PR description sums up the changes and reasons why they should be introduced.
  • I have provided docstrings for the public items that I want to introduce. (No new public API.)
  • I have adjusted the documentation in ./docs/source/. (The ownership proposal remains in docs: define cluster and pool manager ownership #1265.)
  • I added appropriate Fixes: annotations to PR description.

@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

  • Ask an admin to enable usage-based reviews

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 69 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: QUIET
  • Plan: Essentials
  • Run ID: fa78f516-3d4a-453d-b5d9-5a3030eff946
Commits

Reviewing files that changed from the base of the PR and between babdd72 and 7545052.

Files selected for processing: 8

  • cassandra/_lifecycle.py
  • cassandra/_management.py
  • cassandra/cluster.py
  • tests/unit/test_cluster.py
  • tests/unit/test_control_connection.py
  • tests/unit/test_lifecycle_coordination.py
  • tests/unit/test_management_reactors.py
  • tests/unit/test_management_runtime.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@dkropachev
dkropachev force-pushed the lifecycle/1257-reactor-completion branch from 5c09fa6 to 9762d69 Compare November 14, 2026 14:27
@dkropachev
dkropachev force-pushed the lifecycle/1257-runtime-adapters branch from a883895 to dfff849 Compare November 14, 2026 14:27
@dkropachev
dkropachev force-pushed the lifecycle/1257-reactor-completion branch from 9762d69 to e9b3d6a Compare November 14, 2026 15:23
@dkropachev
dkropachev force-pushed the lifecycle/1257-runtime-adapters branch from dfff849 to 24a6696 Compare November 14, 2026 15:23
@dkropachev
dkropachev force-pushed the lifecycle/1257-reactor-completion branch from e9b3d6a to 98078d5 Compare November 14, 2026 16:11
@dkropachev
dkropachev force-pushed the lifecycle/1257-runtime-adapters branch from 24a6696 to ab79ee4 Compare November 14, 2026 16:11
@dkropachev
dkropachev force-pushed the lifecycle/1257-reactor-completion branch from 98078d5 to c7f0c10 Compare November 14, 2026 16:54
@dkropachev
dkropachev force-pushed the lifecycle/1257-runtime-adapters branch from ab79ee4 to 0bf216c Compare November 14, 2026 16:54
@dkropachev
dkropachev force-pushed the lifecycle/1257-reactor-completion branch from c7f0c10 to e190e35 Compare November 14, 2026 19:29
@dkropachev
dkropachev force-pushed the lifecycle/1257-runtime-adapters branch from 0bf216c to 8bf57e5 Compare November 14, 2026 19:29
@dkropachev
dkropachev marked this pull request as ready for review November 14, 2026 20:47
@qodo-scylladb

qodo-scylladb Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (4) 📎 Requirement gaps (2)



🔴 High

1. Host updates can arrive out of order 🐞 Bug
Description
submit_observation() sends an overloaded observation through _defer_observation(), whose
health-slot delivery takes priority over already-admitted ordinary events. When callback capacity
becomes available, a later readiness, location, or pool-result observation can reach listeners
before an earlier lifecycle event still queued on the cluster lane.
Code

cassandra/_lifecycle.py[R570-573]

+        try:
+            self.submit(notification)
+        except ManagementOverloaded:
+            self._defer_observation(host, family, notification)
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Overloaded observations can overtake earlier admitted lifecycle events because their health-slot delivery has priority over the ordinary queue.

## Fix Focus Areas
- cassandra/_lifecycle.py[568-573]
- cassandra/_lifecycle.py[541-566]

## Recommended Fix
Retain bounded overload handling, but sequence deferred positive observations after previously admitted lifecycle events. Add a controlled saturation test that checks listener delivery order.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


2. An overloaded host can stop recovering 🐞 Bug
Description
_submit_pool_creation_work() now enters compatibility_work(), which can reject non-waiting
admission before returning a pool future. During host-UP handling, that exception exits
_on_up_prepared() without calling finish_up() or registering a completion callback, leaving the
host marked as currently handling UP and blocking later UP attempts.
Code

cassandra/cluster.py[R4334-4336]

+        with _lifecycle.compatibility_work(self._management_runtime) as attach:
+            with self._lock:
+                work_future = self.submit(work)
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Pool-work admission can now fail under management load while host-UP handling retains its in-progress flag.

## Fix Focus Areas
- cassandra/cluster.py[4334-4336]
- cassandra/cluster.py[2409-2417]

## Recommended Fix
Ensure the host-UP exception path releases its handling state when pool submission fails, without losing cleanup of any futures already created. Test saturation during host-UP pool submission followed by another UP attempt.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



🟠 Medium

3. Owner thread stalls until caller drains 📎 Requirement gap
Description
deliver in _queue_event calls ready.wait() with no timeout on a management handler thread
before taking the lifecycle lock. It stays blocked until a caller's drain() sets ready, so a
caller that keeps a transaction open holds up other admitted lifecycle work on that runtime.
Code

cassandra/_lifecycle.py[R478-479]

+        def deliver():
+            ready.wait()
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

Issue description
The `deliver` closure created in `LifecycleCoordinator._queue_event` calls `ready.wait()` on a management handler thread. The handler therefore blocks on a future wait until a caller drains, which violates the nonblocking handler requirement.

Fix Focus Areas
- cassandra/_lifecycle.py[478-479]
- cassandra/_lifecycle.py[566-572]

Recommended Fix
Do not block inside the handler. Either have `deliver` return immediately when `ready` is not set and re-queue itself on the caller's drain, or make the caller's `drain()` enqueue the delivery only after the event is ready. Keep the lock ordering that `_queue_event` relies on, and add a deterministic test in which a transaction stays open and a second management handler still makes progress.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


View medium (3)
4. Unknown-host callbacks lost under load 📎 Requirement gap
Description
submit_observation ignores _defer_observation returning False, while the unknown-host path
neither retains the callback nor initiates a refresh beyond setting an unconsumed demand flag. When
admission is saturated and the host is neither current nor retired, readiness and location callbacks
can be lost, and a dropped pool-results completion leaves its waiter unrun.
Code

cassandra/_lifecycle.py[R572-573]

+        except ManagementOverloaded:
+            self._defer_observation(host, family, notification)
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Overloaded observations for unknown hosts lose their callbacks without an explicit failure. Their refresh demand is also never consumed, despite the comment saying it requests a refresh.

## Fix Focus Areas
- cassandra/_lifecycle.py[527-532]
- cassandra/_lifecycle.py[568-573]
- cassandra/_management.py[702-725]

## Recommended Fix
Check `_defer_observation`’s return value in `submit_observation`. If an observation is not retained for a reason other than intentional removal or a closed coordinator, retain its callback or raise `ManagementOverloaded` so the caller can apply its recovery policy. Connect `take_refresh_demand()` to the cluster owner and schedule a node refresh when it returns True; if no refresh is intended, remove the claim that one is requested and log the declined observation at debug level. Add a deterministic test that saturates the runtime for an unknown or stale host and verifies that a completion callback runs or the caller receives an explicit failure, and that any claimed refresh is initiated.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


5. Failed executor handoff can strand a waiting drain 🐞 Bug
Description
When executor.submit(self.drain) fails, submit_async removes the event and sets its ready flag,
but it leaves _draining set and _idle cleared. If another thread's drain() had already seen
that event, cleared _idle and started waiting, the lane's deliver then returns early because
event not in self._events, so nothing sets _idle and that thread blocks forever.
Code

cassandra/_lifecycle.py[R605-609]

            with self._lock:
-                try:
+                if event in self._events:
                    self._events.remove(event)
-                except ValueError:
-                    if self.closed:
-                        return None
-                else:
+                    event[1].set()
+                    return None
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
When `submit_async` withdraws an event after a failed executor handoff, `_idle` stays cleared, so a concurrent `drain()` waits forever.

## Fix Focus Areas
- cassandra/_lifecycle.py[603-614]
- cassandra/_lifecycle.py[478-483]

## Recommended Fix
In the failure branch, after `self._events.remove(event)` (still under `self._lock`), check `if not self._events and not self._active_event:` and if so set `self._draining = False` and call `self._idle.set()`. A second option is to do the same bookkeeping in `deliver`'s early-return path when the event was already withdrawn.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


6. Shut-down clusters can start fresh runtime threads 🐞 Bug
Description
_ensure_runtime() neither checks self._closed nor coordinates runtime installation with
LifecycleCoordinator.close(), so it can start a ManagementRuntime after closure or finish
starting one after shutdown has checked for a runtime to close. Calls through synchronous_api,
Session.__init__, or _defer_observation can trigger this path; even if a pending event is
rejected, the runtime’s four daemon threads and reactor borrow remain until a weakref finalizer
runs.
Code

cassandra/_lifecycle.py[R444-453]

+        with self._runtime_lock:
+            if self._runtime is None:
+                if self._cluster is None:
+                    from cassandra.io.asyncioreactor import AsyncioConnection
+                    connection_class = AsyncioConnection
+                else:
+                    connection_class = self._cluster.connection_class
+                runtime = ManagementRuntime(None)
+                try:
+                    runtime.start()
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`_ensure_runtime()` can start or install a `ManagementRuntime` after coordinator closure, including when first-use startup races with shutdown. The runtime then remains open despite later shutdown or event-rejection checks.

## Fix Focus Areas
- cassandra/_lifecycle.py[249-250]
- cassandra/_lifecycle.py[441-469]
- cassandra/_lifecycle.py[514-527]
- cassandra/_lifecycle.py[998-1009]

## Recommended Fix
Coordinate runtime installation and closure under a shared lock. When no runtime exists and closure has been claimed, do not start one; raise `ManagementClosed` or return `None` and handle it at call sites. If closure occurs during startup, close the newly started runtime. Check closure before `_defer_observation` calls `_ensure_runtime()`, and have `synchronous_api` skip runtime admission for an already shut-down cluster so the wrapped method raises its normal shutdown error. Test shutdown racing with first-use startup.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



Resolved findings
1. Lifecycle drain can hang if the runtime closes first ✓ Resolved

Resolving commit: f0afa17

Resolution reason
At PR head, drain() now waits in a loop with a 0.05s timeout (while not self._idle.wait(0.05):),
and on each iteration checks runtime.closed.is_set(); if the runtime has closed, it clears _events
and _pending_observations, resets _draining, and sets _idle (under self._lock), so the waiting
caller is released. This is the "wait in a loop with a timeout and stop once the runtime reports
closing" alternative from the suggested fix. Also, close() sets every queued event's ready gate and
clears state. The hang scenario is therefore addressed.
Description
The new drain() clears _idle and then blocks on self._idle.wait() with no timeout. Only
deliver() or LifecycleCoordinator.close() ever set _idle again. If the management runtime
closes on its own while events are still queued (for example after it loses its completion reactor),
it throws away queued notifications without running deliver, so the user or executor thread
waiting in drain() never wakes.


Tip of the day
💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗


Powered by Qodo

Comment thread cassandra/_lifecycle.py Outdated
Comment on lines +570 to +573
try:
self.submit(notification)
except ManagementOverloaded:
self._defer_observation(host, family, notification)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High

1. Host updates can arrive out of order 🐞 Bug ≡ Correctness

submit_observation() sends an overloaded observation through _defer_observation(), whose
health-slot delivery takes priority over already-admitted ordinary events. When callback capacity
becomes available, a later readiness, location, or pool-result observation can reach listeners
before an earlier lifecycle event still queued on the cluster lane.
Agent Prompt
## Issue description
Overloaded observations can overtake earlier admitted lifecycle events because their health-slot delivery has priority over the ordinary queue.

## Fix Focus Areas
- cassandra/_lifecycle.py[568-573]
- cassandra/_lifecycle.py[541-566]

## Recommended Fix
Retain bounded overload handling, but sequence deferred positive observations after previously admitted lifecycle events. Add a controlled saturation test that checks listener delivery order.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Comment thread cassandra/cluster.py
Comment on lines +4334 to +4336
with _lifecycle.compatibility_work(self._management_runtime) as attach:
with self._lock:
work_future = self.submit(work)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High

2. An overloaded host can stop recovering 🐞 Bug ☼ Reliability

_submit_pool_creation_work() now enters compatibility_work(), which can reject non-waiting
admission before returning a pool future. During host-UP handling, that exception exits
_on_up_prepared() without calling finish_up() or registering a completion callback, leaving the
host marked as currently handling UP and blocking later UP attempts.
Agent Prompt
## Issue description
Pool-work admission can now fail under management load while host-UP handling retains its in-progress flag.

## Fix Focus Areas
- cassandra/cluster.py[4334-4336]
- cassandra/cluster.py[2409-2417]

## Recommended Fix
Ensure the host-UP exception path releases its handling state when pool submission fails, without losing cleanup of any futures already created. Test saturation during host-UP pool submission followed by another UP attempt.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Serialize callback admission with startup and shutdown, and drain accepted callbacks on the reactor thread when dispatch stops. Reject later submissions after asyncore service loss and remove callbacks whose startup failed.

Close inherited socket and dispatcher descriptors before creating the child reactor without acquiring parent locks. Cover startup failures, shutdown backlogs, loop failure and descriptor ownership with socket-free tests.
Drain libev completions after exit only for the shutdown owner, so a retired
idle runner cannot claim a replacement runner's work. Stop a management
keeper armed concurrently with cleanup or final borrower release.

Separate permanent asyncore management failure from ordinary I/O shutdown.
Drain accepted completions before retiring the failed runner, reject stale
management channels, and allow ordinary connections to restart I/O.

Gate tests at the actual admission, idle-exit and timer-arm boundaries.
Exercise asyncio final draining before queued normal drains can run, and
always stop the chaining test's application loop when management join fails.
Manager shutdown can retire a pool owner before the executor creates its
worker. PyPy can reuse that owner's native thread ID, so comparing saved
integer IDs falsely reports owner-thread disposal.

Retain actual Thread objects and verify worker execution context. Preserve
the exactly-once disposal and released-credit assertions; an exactly-once
owner-disposal negative control still fails the owner-exclusion assertion.
@dkropachev
dkropachev force-pushed the lifecycle/1257-runtime-adapters branch from 8bf57e5 to f0afa17 Compare November 15, 2026 07:57
Deliver existing lifecycle notifications through the bounded management
runtime while retaining publication locks and causal observation ordering.
Reserve synchronous API admission before effects, allow bounded callback
reentry between policy phases, and reject blocking calls on owner or I/O
threads. Keep ordinary query routing on its existing path.

Gate transaction notifications in the scheduler, release nested async
admission through its active drainer, and restore idle state after executor
rejection or runtime loss. Report unknown-host overload explicitly and
release host-UP handling after failed pool admission.

Coordinate lazy runtime installation with shutdown and defer managed
cleanup until physical pool work and the completion barrier retire. Cover
saturation, reentry, startup, shutdown, nested recovery and collection with
controlled interleavings on rebuilt Cython and pure Python runtimes.
@dkropachev
dkropachev force-pushed the lifecycle/1257-runtime-adapters branch from f0afa17 to 7545052 Compare November 15, 2026 07:59
@dkropachev
dkropachev force-pushed the lifecycle/1257-reactor-completion branch from 1ab063f to 1a2abdf Compare November 15, 2026 12:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant