Predicate-aware HNSW traversal: filtered vector search with user-function and RBAC participation

[kriszyp]

Implements #1241 — predicate-aware HNSW traversal so a vector sort combined with a filter keeps exploring until it has ef matching results, instead of post-filtering a fixed top-ef candidate set (which under-fills under selective filters and quietly leaks the presence of nearby restricted records).

What changed

• HierarchicalNavigableSmallWorld.search(cond, ctx, filter?) accepts an optional (primaryKey) => boolean predicate. It gates result admission at layer 0 only; upper layers and the candidate/visited sets ignore it, so non-matching nodes still route (ACORN). A visit budget ef * filterExpansion (default 24) bounds work and returns partial results on exhaustion; nodesVisited / filterEvaluations are exposed for tuning.
• search.ts (executeConditions) composes companion AND conditions + a request vectorFilter + a static allowReadRecord RBAC hook into one predicate and pushes it into a leading filterable custom index; it also stays in the post-filter chain as a deterministic re-check. Very selective condition filters continue to take the exact brute-force path via the planner's existing count-ordering.
• Table.search plumbs vectorFilter (JS-API only — never parsed from a REST query string) and record-level allowReadRecord.
• Unit tests (index-level mechanism + end-to-end vectorFilter/RBAC/int8/OR), a filtered-recall benchmark scenario, and a DESIGN.md note.

Where to look / lower-confidence areas

• filterExpansion default = 24, not the design sketch's 8. Harper's HNSW visits a large fraction of the graph per query, so filling ef at selectivity s needs ~ef/s visits; 8 truncated recall below post-filtering at ≤12% selectivity. 24 fills down to ~4% and — per the new benchmark — predicate recall dominates post-filter at every selectivity (e.g. 99.3% vs 23.7% at 1%), with brute-force the fast crossover at 0.1%. This budget/selectivity interaction with the planner's brute-force threshold is the main tuning knob worth a second opinion.
• executeConditions reorders sibling-filter construction before executing the lead only when the lead is a filterable custom index (HNSW), which never populates the join filtered map. All other paths (joins etc.) keep the original ordering. Covered by the existing query/join suites.
• RBAC record guards run at the same processEntry filter stage as every existing WHERE-clause filter (against the stored record, before any source revalidation). This means allowReadRecord inherits the same cache/source-staleness semantics that condition-based filtering already has on caching tables — noted here as a pre-existing property, not introduced by this PR; defense-in-depth re-checking post-revalidation would be a separate, filter-wide hardening.

Review

Cross-model review (Gemini + Codex + Harper-domain adjudication) run pre-PR. Addressed in the diff: OR-query RBAC enforcement, always-active visit budget (bounds the filled-but-distant case), no mutation of the caller's condition object, and guards built once at top level.

Generated by KrAIs (Claude Opus 4.8).

Docs: companion documentation PR: HarperFast/documentation#580 (Filtered vector search docs).

[claude]

Reviewed; no blockers found.