Skip to content

Predicate-aware HNSW traversal: filtered vector search with user-function and RBAC participation #1768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kriszyp merged 3 commits into from
Jul 13, 2026
Merged

Predicate-aware HNSW traversal: filtered vector search with user-function and RBAC participation #1768

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b2f473b
Predicate-aware HNSW traversal: filtered vector search with RBAC (#1241)
kriszyp Jul 12, 2026
b8f36c7
fix: prettier reflow of DESIGN.md table
kriszyp Jul 12, 2026
dfc4a6e
review: default filterState for direct searchLayer callers, null-guar…
kriszyp Jul 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions benchmarks/hnsw-search.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,96 @@ console.log(` Latency: ${fmtNs(avgNs)}/query avg`);
console.log(` Throughput: ${qps.toFixed(0)} qps`);
console.log(` Nodes visited/query: ${(hnsw.nodesVisitedCount / N_QUERIES).toFixed(1)}`);
console.log(` Recall@${TOP_K}: ${(recall * 100).toFixed(1)}%\n`);

// ---------------------------------------------------------------------------
// Filtered-recall scenario (#1241): recall@K vs. selectivity for three strategies —
// post-filter : plain search, then drop non-matching from the fixed top-ef set (today's behavior)
// predicate : predicate-aware traversal (filter passed into search, ACORN-style)
// brute-force : exact distance over the matching set only (the selective-filter fallback)
// Predicate traversal should dominate post-filter on recall at low selectivity without pathological
// latency; brute-force is the crossover once the matching set is tiny.
// ---------------------------------------------------------------------------

const idxOf = (key) => parseInt(key.slice(1));
const SELECTIVITIES = [0.5, 0.1, 0.01, 0.001];

console.log(`Filtered recall@${TOP_K} vs. selectivity (${N_QUERIES} queries):\n`);
console.log(' selectivity matches post-filter predicate brute-force pred.lat bf.lat');
console.log(' ' + '-'.repeat(88));

for (const p of SELECTIVITIES) {
// Deterministic-ish membership: mark a p-fraction of vectors as matching the companion predicate.
const matches = new Array(N_VECTORS);
let matchCount = 0;
for (let i = 0; i < N_VECTORS; i++) {
matches[i] = Math.random() < p;
if (matches[i]) matchCount++;
}
if (matchCount === 0) {
console.log(
` ${(p * 100).toFixed(1).padStart(9)}% ${String(matchCount).padStart(7)} (no matching vectors — skipped)`
);
continue;
}
const filter = (key) => matches[idxOf(key)];
const k = Math.min(TOP_K, matchCount);

// Ground truth: the k nearest among the matching vectors only.
const filteredGT = queries.map((query) => {
const scored = [];
for (let i = 0; i < N_VECTORS; i++) if (matches[i]) scored.push({ i, d: cosineDistance(query, vectors[i]) });
scored.sort((a, b) => a.d - b.d);
return new Set(scored.slice(0, k).map((x) => x.i));
});

let postRecall = 0;
let predRecall = 0;
let bfRecall = 0;
let predNs = 0;
let bfNs = 0;

for (let q = 0; q < N_QUERIES; q++) {
const query = queries[q];
const gt = filteredGT[q];

// post-filter: unfiltered search, then keep matching, take k
const plain = hnsw.search({ target: query, comparator: 'sort', descending: false }, {});
const postIdx = new Set(
plain
.filter((r) => matches[idxOf(r.key)])
.slice(0, k)
.map((r) => idxOf(r.key))
);
postRecall += overlap(gt, postIdx) / k;

// predicate-aware traversal
const t0 = performance.now();
const pred = hnsw.search({ target: query, comparator: 'sort', descending: false }, {}, filter);
predNs += (performance.now() - t0) * 1e6;
const predIdx = new Set(pred.slice(0, k).map((r) => idxOf(r.key)));
predRecall += overlap(gt, predIdx) / k;

// brute-force over the matching set
const b0 = performance.now();
const scored = [];
for (let i = 0; i < N_VECTORS; i++) if (matches[i]) scored.push({ i, d: cosineDistance(query, vectors[i]) });
scored.sort((a, b) => a.d - b.d);
bfNs += (performance.now() - b0) * 1e6;
const bfIdx = new Set(scored.slice(0, k).map((x) => x.i));
bfRecall += overlap(gt, bfIdx) / k;
}

const pct = (x) => `${((x / N_QUERIES) * 100).toFixed(1)}%`.padStart(9);
console.log(
` ${(p * 100).toFixed(1).padStart(9)}% ${String(matchCount).padStart(7)} ` +
`${pct(postRecall).padStart(11)} ${pct(predRecall).padStart(9)} ${pct(bfRecall).padStart(11)} ` +
`${fmtNs(predNs / N_QUERIES).padStart(8)} ${fmtNs(bfNs / N_QUERIES).padStart(8)}`
);
}
console.log('');

function overlap(a, b) {
let n = 0;
for (const x of a) if (b.has(x)) n++;
return n;
}
29 changes: 15 additions & 14 deletions resources/DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,20 +75,21 @@ One giant `makeTable()` factory that returns a `TableResource extends Resource`

## "Where is X" cheat sheet

| Question | Where |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| How is a CRUD request authorized? | `Table.ts → #section: authz-hooks`; defaults in `Resource.ts` (`allowRead` etc.) |
| Where does versioning / conflict resolution happen? | `Table.ts → _writeUpdate` (`#section: write-path-internals`) |
| How does `search()` choose an index? | `Table.ts → search` (`#section: search-query`) |
| How are subscriptions replayed? | `Table.ts → subscribe` (`#section: pub-sub`) |
| How is the response body shaped (select clause)? | `Table.ts → transformEntryForSelect` (`#section: search-query`) |
| Where is record-level TTL evaluated? | `Table.ts → setTTLExpiration` (`#section: lifecycle-admin`); `Updatable.getExpiresAt` (`#section: setup-and-factory`) |
| How are residencies enforced (replication)? | `Table.ts → #section: lifecycle-admin` (residency block: `getResidencyRecord`, `setResidency`, `setResidencyById`, `getResidency`) |
| How is the RecordObject prototype applied? | `RecordEncoder.ts` (see `../DESIGN.md`) |
| Where is the per-request transaction stored? | `transaction.ts` + `contextStorage` (AsyncLocalStorage) |
| How does a query opt out of a read snapshot? | Pass `snapshot: false` on the search request (e.g. `get_analytics`). `Table.ts → search` calls `txn.useReadTxn(snapshot === false)`; on RocksDB `DatabaseTransaction.getReadTxn` then builds the read txn with `{ disableSnapshot: true }` so a long scan reads latest without pinning a snapshot. No-op on LMDB (`LMDBTransaction.useReadTxn`). |
| How does a URL path map to a Resource? | `Resources.ts → getMatch` (exact/prefix fast path) then `matchParamRoute` (parameterised routes); see "Path routing" below |
| How does HNSW keep the graph connected on delete? | `indexes/HierarchicalNavigableSmallWorld.ts → index()` delete path: zero-degree orphans reindexed via `needsReindexing`; severed multi-node islands detected and reconnected by `repairSeveredNeighbors` (#1712) |
| Question | Where |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| How is a CRUD request authorized? | `Table.ts → #section: authz-hooks`; defaults in `Resource.ts` (`allowRead` etc.) |
| Where does versioning / conflict resolution happen? | `Table.ts → _writeUpdate` (`#section: write-path-internals`) |
| How does `search()` choose an index? | `Table.ts → search` (`#section: search-query`) |
| How are subscriptions replayed? | `Table.ts → subscribe` (`#section: pub-sub`) |
| How is the response body shaped (select clause)? | `Table.ts → transformEntryForSelect` (`#section: search-query`) |
| Where is record-level TTL evaluated? | `Table.ts → setTTLExpiration` (`#section: lifecycle-admin`); `Updatable.getExpiresAt` (`#section: setup-and-factory`) |
| How are residencies enforced (replication)? | `Table.ts → #section: lifecycle-admin` (residency block: `getResidencyRecord`, `setResidency`, `setResidencyById`, `getResidency`) |
| How is the RecordObject prototype applied? | `RecordEncoder.ts` (see `../DESIGN.md`) |
| Where is the per-request transaction stored? | `transaction.ts` + `contextStorage` (AsyncLocalStorage) |
| How does a query opt out of a read snapshot? | Pass `snapshot: false` on the search request (e.g. `get_analytics`). `Table.ts → search` calls `txn.useReadTxn(snapshot === false)`; on RocksDB `DatabaseTransaction.getReadTxn` then builds the read txn with `{ disableSnapshot: true }` so a long scan reads latest without pinning a snapshot. No-op on LMDB (`LMDBTransaction.useReadTxn`). |
| How does a URL path map to a Resource? | `Resources.ts → getMatch` (exact/prefix fast path) then `matchParamRoute` (parameterised routes); see "Path routing" below |
| How does HNSW keep the graph connected on delete? | `indexes/HierarchicalNavigableSmallWorld.ts → index()` delete path: zero-degree orphans reindexed via `needsReindexing`; severed multi-node islands detected and reconnected by `repairSeveredNeighbors` (#1712) |
| How is a filter applied _during_ a vector search? | Predicate-aware traversal (#1241): `search.ts → executeConditions` composes companion AND conditions + a request `vectorFilter` + a static `allowReadRecord` RBAC hook into one `(primaryKey) => boolean` (`composeRecordFilter`) and passes it to `HierarchicalNavigableSmallWorld.search(cond, ctx, filter)`. The filter gates result admission at layer 0 only (routing ignores it, ACORN-style); a visit budget (`filterExpansion`) bounds the under-filled/selective case. Very selective _condition_ filters are instead diverted to the exact brute-force path by the query planner's `estimateCountAsSort` ordering. |

---

Expand Down
Loading