fix(cluster): exclude shed noise points from cluster membership

[rdmmf]

Rebased onto current dev (supersedes #14, which was branched off a stale dev).

Problem

Two complaints, one root cause:

1. Cluster metadata wrong — member_count, cohesion_score inflated.
2. Big low-cohesion clusters instead of small precise ones — e.g. 3 binaries, 2 of them 100% similar, land in one ~80%-cohesion cluster of 3 instead of a precise pair.

Root cause

HDBSCAN keeps a tight core under one condensed-tree node while loosely-attached members peel off one-at-a-time as singleton noise (each < min_cluster_size), so no split node is created. The code then built each node's membership by walking every descendant leaf, including those shed singletons — diluting the tight core.

Not a parameter problem: min_samples=1 (needed to keep identical binaries tight) is already the default.

Fix

A leaf belongs to an ancestor cluster C only if it survives to C's death lambda (fallout(leaf) >= death(C)). Shed points become noise.

• Genuine sub-structure preserved: parent + tight child clusters still all emitted.
• Dendrogram unchanged — tree/tree_links storage untouched; only membership/metadata changed.
• Excludes the synthetic global root (was a giant ~0-cohesion cluster) and drops <min_cluster_size survivors.
• direct_members derived from each leaf's deepest surviving cluster.

Shared logic in cluster_common.hierarchical_membership, applied to all hierarchical callers: run_clustering (function + binary) and run_pool_bin_clustering. Redis-free self-check: uv run python bsimvis/app/services/cluster_common.py.

Testing

• Membership logic verified against real HDBSCAN condensed trees (self-check asserts tight-pair-vs-loose and two-family cases).
• NOT run end-to-end against live kvrocks (no access to real sim data) — recommend a re-cluster on a throwaway collection before undrafting.

🤖 Generated with Claude Code

[rdmmf]

Added: recursive-Leiden backend (scope A)

Beyond the shed-noise membership fix, this branch now adds a selectable Leiden clustering backend for the function path.

Why: HDBSCAN's single-linkage buries high-cohesion sub-groups inside low-density clusters — a 14-func cluster @ cohesion 0.54 had 14 leaf children and zero sub-clusters, so the 100% groups inside were invisible. No HDBSCAN param surfaces them (the tree genuinely has no sub-nodes). On the real null data, 6 of 52 clusters hid a high-cohesion sub-group.

What: clustering.algorithm = "leiden" runs recursive community detection on the sparse similarity graph — keeps low-cohesion parents as dendrogram levels, surfaces tight sub-groups as their children, stop_cohesion prevents over-splitting already-tight groups. Graph-native (missing pairs absent, not distance-1).

Verified read-only on real data: the 0.54 blob preserved as a parent with 0.96–1.0 sub-groups surfaced beneath it; aggregate 221 clusters @ avg cohesion 0.982; re-clusters one touched component in ~3ms.

Implementation: build_leiden_tree emits the same tree_df shape HDBSCAN produces → flows through the existing persistence/metadata path unchanged. HDBSCAN tree-build extracted verbatim into _hdbscan_tree; run_clustering dispatches on algorithm. HDBSCAN stays default. Deps: python-igraph/leidenalg.

Not yet: full write-path run on live data (would clear existing clusters — recommend a throwaway-collection re-cluster before undrafting); binary path still HDBSCAN-only (follow-up).