Skip to content

fix(brain): destroy retired skill instances' ROS entities on reload (navigator leak) - #505

Merged
theo-michel merged 3 commits into
mainfrom
theo/skills-reload-navigator-leak
Jul 8, 2026
Merged

fix(brain): destroy retired skill instances' ROS entities on reload (navigator leak)#505
theo-michel merged 3 commits into
mainfrom
theo/skills-reload-navigator-leak

Conversation

@theo-michel

Copy link
Copy Markdown
Contributor

Problem

Every skill reload leaks ROS entities. Two compounding paths, neither with teardown:

  1. Double instantiation per load — discovery calls SkillLoader._get_name(), which builds a throwaway instance of each skill class just to read .name; the catalog then builds the kept one. For navigate_to_position each instance is 3 BasicNavigator nodes (~10 action clients each). The sim skill's throwaway adds another navigator even in real-robot mode (it's only dropped after discovery, in _apply_sim_swap).
  2. No retirement on reloadreload_all() / reload_selective() overwrite the catalog dicts, orphaning old instances. Orphans are cyclic garbage (Node → subscription → bound callback → Node), so their zenoh graph entities stay registered until a gen-2 GC pass — which an idle process may not run for hours.

Measured on the live robot (mars-the-44th, 2026-07-07)

  • 8 duplicate navigator entity sets (~90 subscriptions) alive in the skills server across /, /mapfree, /navigation after an 83-reload storm (a client hammered /brain/reload at ~100 Hz for 30 min — separate bug, tracked separately).
  • One triggered /brain/reload_primitives: +2 entity sets in /mapfree & /navigation, +3 in root (sim throwaway), +19.5 MB RSS, 39.5 s duration.
  • Isolated repro: orphans accumulate across reload cycles; a single gc.collect() releases all of them — nothing hard-pins them, they're just uncollected cyclic garbage.

Fix

  • Skill.shutdown() lifecycle hook (no-op default). Deliberately leaves shared-server-node entities alone — destroying entities under a spinning executor is unsafe (fix(brain): stop destroying ROS entities under spinning executors (skills-server cancel crash-loop) #497).
  • Nav2Controller.destroy() / SimPathPlanningController.destroy() — destroy the navigator nodes, including assisted_teleop_client, which Humble's BasicNavigator.destroy_node() misses and whose live handle keeps the rcl node registered (destroy_when_not_in_use).
  • SkillLoader._get_name — shuts the throwaway instance down.
  • SkillRepository — retires replaced/pruned instances on reload_all, reload_selective, and _prune_stale_skills.
  • SkillsActionServer — disposal gate: retired instances are destroyed immediately when idle, or deferred to execute_callback's finally while a skill runs. A mid-run reload can retire the currently executing instance, whose entities its execute() is still spinning; destroying those live would recreate the fix(brain): stop destroying ROS entities under spinning executors (skills-server cancel crash-loop) #497 crash class. Navigator nodes are never attached to any executor when no skill is executing, so idle-time destruction is exactly as safe as the GC destruction that already happens today — just deterministic.

Verification

  • With gc.disable(), three simulated reload cycles against the live graph hold at exactly 1 entity set (previously +2/cycle) and drop to 0 on final retire, including /assisted_teleop/_action/* and the sim skill's nodes.
  • Fast no-ROS pytest bucket (test_backwards_compat.py, 12 loader/discovery tests) passes with the patched loader — the new shutdown() call in _get_name is exercised by every discovery test.
  • py_compile clean on all six files; 120-col ruff limit respected.

Note for deploy: brain_client needs a rebuild + skills-server restart; the two workspace/innate_skills/ files hot-reload on their own afterward.

…navigator leak)

Every skill reload built each code skill twice (a throwaway instance in
SkillLoader._get_name plus the kept one) and dropped the previous kept
instance with no teardown. Dropped instances are cyclic garbage (Node ->
subscription -> bound callback -> Node), so their ROS entities stayed in
the zenoh graph until a rare gen-2 GC pass. For navigate_to_position that
is 3 BasicNavigator nodes (~10 action clients each) per instance: the live
box showed 8 duplicate subscriber sets on every */_action/feedback|status
topic across /, /mapfree and /navigation after an 83-reload storm, ~20 MB
RSS growth per reload, and 30-40 s reload latency.

Fix, verified against the live graph with GC disabled (entity count now
stays at exactly one set across reload cycles and drops to zero on final
retire):

- Skill.shutdown(): lifecycle hook for skill-owned ROS entities. Entities
  on the shared server node are deliberately left alone (see #497 -- no
  destroys under a spinning executor).
- Nav2Controller/SimPathPlanningController.destroy(): destroy the
  navigator nodes, including assisted_teleop_client, which Humble's
  BasicNavigator.destroy_node() misses and whose live handle would keep
  the rcl node registered.
- SkillLoader._get_name: shut the throwaway instance down.
- SkillRepository: retire replaced/pruned instances on reload_all,
  reload_selective and _prune_stale_skills.
- SkillsActionServer: disposal gate -- retired instances are destroyed
  immediately when idle, or deferred to execute_callback's finally while
  a skill is running (a mid-run reload may retire the running instance,
  whose entities its execute() is still spinning).
@greptile-apps

greptile-apps Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds deterministic cleanup for skill reloads. The main changes are:

  • Adds a Skill.shutdown() lifecycle hook.
  • Retires replaced skill instances during reloads and pruning.
  • Defers retired-instance disposal while a skill is running.
  • Destroys navigator-owned ROS nodes during shutdown.
  • Cleans up temporary discovery instances after reading skill names.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
workspace/innate_skills/navigate_to_position.py Adds shutdown cleanup for the real navigation controller and its navigator nodes.
workspace/innate_skills/navigate_to_position_sim.py Adds shutdown cleanup for the sim navigator and path-planning node.
ros2_ws/src/brain/brain_client/brain_client/nodes/skills_server.py Adds the disposal gate for retired skill instances during skill execution.
ros2_ws/src/brain/brain_client/brain_client/skills/catalog.py Retires replaced and pruned code-skill instances during reload operations.
ros2_ws/src/brain/brain_client/brain_client/skills/loader.py Shuts down temporary skill instances created during name discovery.
ros2_ws/src/brain/brain_client/brain_client/skills/types.py Adds the base shutdown hook for skill-owned resource cleanup.

Reviews (3): Last reviewed commit: "fix: don't let a failed assisted_teleop ..." | Re-trigger Greptile

Comment thread workspace/innate_skills/navigate_to_position.py
Comment thread workspace/innate_skills/navigate_to_position_sim.py
Comment thread workspace/innate_skills/navigate_to_position.py
Comment thread workspace/innate_skills/navigate_to_position_sim.py
@theo-michel
theo-michel merged commit 9b0d53e into main Jul 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant