Repository navigation
lifecycle: adapt management producers and synchronous APIs - #1269
dkropachev wants to merge 5 commits into
Conversation
|
Warning Review limit reachedEnable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Next included review available in 20 minutes. View limit details
|
5c09fa6 to
9762d69
Compare
a883895 to
dfff849
Compare
9762d69 to
e9b3d6a
Compare
dfff849 to
24a6696
Compare
e9b3d6a to
98078d5
Compare
24a6696 to
ab79ee4
Compare
98078d5 to
c7f0c10
Compare
ab79ee4 to
0bf216c
Compare
c7f0c10 to
e190e35
Compare
0bf216c to
8bf57e5
Compare
Code Review by Qodo
🔴 High 1. Host updates can arrive out of order
|
| try: | ||
| self.submit(notification) | ||
| except ManagementOverloaded: | ||
| self._defer_observation(host, family, notification) |
There was a problem hiding this comment.
🔴 High
1. Host updates can arrive out of order 🐞 Bug ≡ Correctness
submit_observation() sends an overloaded observation through _defer_observation(), whose health-slot delivery takes priority over already-admitted ordinary events. When callback capacity becomes available, a later readiness, location, or pool-result observation can reach listeners before an earlier lifecycle event still queued on the cluster lane.
Agent Prompt
## Issue description
Overloaded observations can overtake earlier admitted lifecycle events because their health-slot delivery has priority over the ordinary queue.
## Fix Focus Areas
- cassandra/_lifecycle.py[568-573]
- cassandra/_lifecycle.py[541-566]
## Recommended Fix
Retain bounded overload handling, but sequence deferred positive observations after previously admitted lifecycle events. Add a controlled saturation test that checks listener delivery order.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| with _lifecycle.compatibility_work(self._management_runtime) as attach: | ||
| with self._lock: | ||
| work_future = self.submit(work) |
There was a problem hiding this comment.
🔴 High
2. An overloaded host can stop recovering 🐞 Bug ☼ Reliability
_submit_pool_creation_work() now enters compatibility_work(), which can reject non-waiting admission before returning a pool future. During host-UP handling, that exception exits _on_up_prepared() without calling finish_up() or registering a completion callback, leaving the host marked as currently handling UP and blocking later UP attempts.
Agent Prompt
## Issue description
Pool-work admission can now fail under management load while host-UP handling retains its in-progress flag.
## Fix Focus Areas
- cassandra/cluster.py[4334-4336]
- cassandra/cluster.py[2409-2417]
## Recommended Fix
Ensure the host-UP exception path releases its handling state when pool submission fails, without losing cleanup of any futures already created. Test saturation during host-UP pool submission followed by another UP attempt.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Serialize callback admission with startup and shutdown, and drain accepted callbacks on the reactor thread when dispatch stops. Reject later submissions after asyncore service loss and remove callbacks whose startup failed. Close inherited socket and dispatcher descriptors before creating the child reactor without acquiring parent locks. Cover startup failures, shutdown backlogs, loop failure and descriptor ownership with socket-free tests.
e190e35 to
8008e60
Compare
Drain libev completions after exit only for the shutdown owner, so a retired idle runner cannot claim a replacement runner's work. Stop a management keeper armed concurrently with cleanup or final borrower release. Separate permanent asyncore management failure from ordinary I/O shutdown. Drain accepted completions before retiring the failed runner, reject stale management channels, and allow ordinary connections to restart I/O. Gate tests at the actual admission, idle-exit and timer-arm boundaries. Exercise asyncio final draining before queued normal drains can run, and always stop the chaining test's application loop when management join fails.
Manager shutdown can retire a pool owner before the executor creates its worker. PyPy can reuse that owner's native thread ID, so comparing saved integer IDs falsely reports owner-thread disposal. Retain actual Thread objects and verify worker execution context. Preserve the exactly-once disposal and released-credit assertions; an exactly-once owner-disposal negative control still fails the owner-exclusion assertion.
8bf57e5 to
f0afa17
Compare
Deliver existing lifecycle notifications through the bounded management runtime while retaining publication locks and causal observation ordering. Reserve synchronous API admission before effects, allow bounded callback reentry between policy phases, and reject blocking calls on owner or I/O threads. Keep ordinary query routing on its existing path. Gate transaction notifications in the scheduler, release nested async admission through its active drainer, and restore idle state after executor rejection or runtime loss. Report unknown-host overload explicitly and release host-UP handling after failed pool admission. Coordinate lazy runtime installation with shutdown and defer managed cleanup until physical pool work and the completion barrier retire. Cover saturation, reentry, startup, shutdown, nested recovery and collection with controlled interleavings on rebuilt Cython and pure Python runtimes.
f0afa17 to
7545052
Compare
1ab063f to
1a2abdf
Compare
Move existing lifecycle notification delivery onto the bounded management runtime while preserving the publication locks and pool recovery ordering. Bind the runtime lazily to the selected reactor and detach its channel after the final cleanup barrier.
DOWN/REMOVE producers retain coalesced negative observations when ordinary I/O or callback credits are full. Deferred positive observations follow previously admitted events and retain causal family ordering. Unknown-host overload is explicit. Transaction readiness is handled by the scheduler; delivery handlers do not wait for the transaction caller. Nested asynchronous recovery joins the active drainer, and failed executor handoffs restore idle state.
Synchronous connect, refresh and keyspace APIs wait for admission before side effects. Callback reentry uses a separate aggregate and bounded prerequisite window. Policies run between released compatibility phases. Blocking APIs reject actual reactor and owner threads. Management-context shutdown defers legacy waits to the existing executor and retains cleanup through the final barrier, including late results. First-use startup cannot install a runtime after shutdown, and failed pool admission releases host-UP handling so recovery can retry.
Fixes #1257. Final part of the stack, dependent on #1268 and #1267, with #1255 beneath them. The ownership contract proposed in #1265 and the prerequisite PRs remain under review. Actual cluster/pool field migration remains in later issues. Ordinary query routing gains no mailbox hop and protocol behavior is unchanged. Workload and performance-budget agreement and benchmark measurements remain outstanding.
Validation on the final source tree: 1,102 tests and 56 subtests pass with rebuilt Cython modules, and the same 1,102 tests and 56 subtests pass with pure-Python core modules. Both configurations also pass all 42 asyncio reactor tests. The broad suites include real libev and Python 3.11 asyncore coverage. PyPy passes 1,054 tests and 56 subtests plus the 42 asyncio tests; 18 native-libev cases are explicitly deselected because that extension is unavailable on PyPy here. Controlled regressions cover saturation, synchronous callback reentry, 128-pool fanout with one executor worker, raw reactor API rejection, managed shutdown, startup unwind, nested recovery and collection.
TZ=UTCis used consistently andgit diff --checkpasses. Live-cluster integration and performance measurements remain outstanding.Pre-review checklist
./docs/source/. (The ownership proposal remains in docs: define cluster and pool manager ownership #1265.)Fixes:annotations to PR description.