Prevent stale device lifecycle commits after removal

[dmulcahey]

Removing a device while pairing or reinterview is still running can publish a later DeviceFullInitEvent or recreate wrapper state after removal. Cancellation alone does not stop an operation that is still unwinding or performing asynchronous cleanup.

Serialize each device's initialization, reinterview, and removal with an ownership generation and a per-device lock. Removal revokes the old generation, waits for its exact task, and only removes the captured wrapper. Same-IEEE rejoin waits for removal; delayed callbacks cannot delete a replacement. Shutdown quiesces lifecycle work before wrapper/controller teardown. Device identity tracking uses weak references.

Removal-event listener failures now still run wrapper teardown; a failed removal does not poison a subsequent rejoin. Tests cover the public removal/event ordering, cancellation-resistant initialization, factory-changing reinterview, immediate rejoin, stale callbacks, cleanup failure, and shutdown. A hook that never finishes cancellation can still delay removal/shutdown; this PR does not add a timeout or abandon live work.

Validation against dev at 66603431339afe37fa0048b70ff31d77dceb8f95: Python 3.12 full suite, 1402 passed, coverage above the 95% project gate; full pre-commit (codespell, Ruff, formatting, mypy, lock check) passed. Relevant regression checks fail on the unchanged base. GitHub CI for Python 3.12/3.13/3.14 is reported separately on the PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.20%. Comparing base (6660343) to head (30b5882).

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #690      +/-   ##
==========================================
+ Coverage   97.19%   97.20%   +0.01%     
==========================================
  Files          57       57              
  Lines       10560    10673     +113     
==========================================
+ Hits        10264    10375     +111     
- Misses        296      298       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Improves ZHA gateway lifecycle robustness by adding per-device lifecycle generations/locks and by quiescing device lifecycle work during shutdown/removal, reducing race conditions between initialization, reinterview, removal, and shutdown.

Changes:

• Introduces per-device lifecycle generation tracking + locks to serialize initialization/reinterview/removal and prevent stale callbacks from committing.
• Refactors device init/reinterview/removal flows to cancel superseded work, await cancellation-resistant tasks appropriately, and harden shutdown teardown ordering.
• Expands gateway tests to cover lifecycle races (init vs remove, remove vs immediate rejoin, shutdown vs in-flight init).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
zha/application/gateway.py	Adds device lifecycle generation/locking + quiesce logic; refactors init/reinterview/removal/shutdown to avoid lifecycle races.
tests/test_gateway.py	Adds/updates tests validating the new lifecycle serialization and shutdown/removal race handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zigpy-review-bot]

Reviewed the lifecycle machinery end-to-end and it holds up well — the generation counter (advanced before cancellation) + per-IEEE lock + removal-task barrier + post-await revalidation is a sound and internally consistent design, and the wait-before-lock ordering keeps the task wait-graph acyclic, so no lock-cycle deadlock is reachable. Test coverage maps cleanly onto each scenario in the description; locally all test_gateway.py tests pass with no wall-clock outliers under looptime, in-venv mypy is clean, and CI is green across 3.12–3.14.

I traced the paths this PR is most exposed on and didn't find a remaining race: an old removal can't clobber a newer lifecycle (the wrapper-identity check in _async_remove_device gates the del self._devices[ieee]), the done-callbacks only delete entries that still point at their own task, and shutdown's quiesce awaits (rather than races) removals before crossing the locks. An independent second-opinion pass reached the same conclusion.

No blockers — just a few optional observations:

• _device_lifecycle_zigpy_devices keeps a strong reference to each removed device's zigpy Device until the gateway is reinitialized (gateway.py:834 is read-only; the dict is only cleared in _async_initialize). It's bounded by distinct IEEEs so it's not a runaway leak, and I understand keeping the generations/locks for the run is deliberate for stale-callback rejection — but this dict retains the heaviest objects (endpoints/clusters/listeners). A weakref, or pruning just this dict on a successful removal, would trim the retention without weakening the ownership checks. Optional.
• Optional readability: _async_remove_device awaits the previous task with a plain asyncio.gather(..., return_exceptions=True) (gateway.py:889) while _async_run_device_lifecycle_operation shields the equivalent wait (:505). The asymmetry is safe (removal tasks are only cancelled at the final super().shutdown(), after quiesce has already awaited them), but a one-line comment on why removal doesn't need the shield would save the next reader the trace.
• Nit: in _async_run_device_lifecycle_operation, await asyncio.wait({previous_removal_task}) would be a simpler equivalent to the shield(gather(...)) wait — same don't-propagate / don't-cancel-the-inner semantics.

Nice work — the invariant is clearly stated and the tests back it up.

[dmulcahey]

Follow-up on the optional observations, addressed in 2a16b7f:

• Strong device retention: valid and corrected. The lifecycle identity map now uses weak values. A pending lifecycle task or active ZHA wrapper still owns the current zigpy device strongly, so same-IEEE stale-callback rejection remains intact, while the identity map no longer keeps removed endpoint, cluster, and listener graphs alive by itself. A focused regression test verifies that an otherwise unowned tracked zigpy device is collectable and that its weak map entry disappears.
• Removal wait asymmetry: the behavior is intentionally unchanged and now documented at the wait site. Initialization and reinterview tasks can be superseded and cancelled, so their wait on an older removal must be shielded. Removal tasks are chained rather than superseded by lifecycle callbacks, so their previous-removal wait does not need that shield.
• asyncio.wait simplification: not applied because it is not equivalent here. asyncio.wait does not retrieve the completed task outcome. The current shield plus gather with return_exceptions consumes a failed removal outcome without letting it poison the replacement lifecycle and without producing an unhandled task exception.