From 709d6c15449f0ecc5b2308965a74fc8d88ae01ef Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 12 Jul 2026 09:24:25 -0600 Subject: [PATCH 1/5] Document filtered vector search: predicate-aware HNSW traversal, vectorFilter, allowReadRecord, filterExpansion (5.2) Companion to HarperFast/harper#1768. Co-Authored-By: Claude Fable 5 --- reference/database/schema.md | 55 +++++++++++++++++++++++++++++++++ release-notes/v5-lincoln/5.2.md | 6 ++++ 2 files changed, 61 insertions(+) diff --git a/reference/database/schema.md b/reference/database/schema.md index 5b342f06..6dfa2517 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -507,6 +507,60 @@ let results = Document.search({ }); ``` + — Conditions combined with a vector sort are evaluated _during_ graph traversal (predicate-aware search): the search keeps exploring until it has enough _matching_ nearest neighbors, instead of finding the nearest candidates first and then dropping the ones that fail the filter. With a selective filter this is the difference between a full result set and an under-filled one. When a companion condition is very selective, Harper instead computes exact distances over just the records matching that condition, which is both exact and faster than traversing the graph. + +### Filtered Vector Search with a Function Predicate + + + +A `vectorFilter` function on the query participates in the traversal the same way, for predicates that are not expressible as conditions: + +```javascript +let results = Document.search( + { + sort: { attribute: 'textEmbeddings', target: searchVector }, + vectorFilter: (record) => record.tenantId === context.user.tenantId && record.status === 'published', + limit: 10, + }, + context +); +``` + +`vectorFilter` is available from the JavaScript API only (it cannot be expressed in a REST query string). The function must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). Records passed to it are frozen. + +### Record-Level Access Control in Vector Search + + + +A resource class can define a static `allowReadRecord(user, record)` check that filters query results per record. For vector queries it participates in the traversal, so a restricted user receives the k nearest records _they are allowed to see_ rather than "nearest k, minus redacted" (which under-fills results and reveals that nearby restricted records exist): + +```javascript +export class Reports extends tables.Reports { + static allowReadRecord(user, record) { + return record.ownerId === user.id || user.role?.permissions?.super_user; + } +} +``` + +`allowReadRecord` applies to all query results from `search()` (vector or not; on caching tables it is enforced against the record actually returned, after any source revalidation). It is a query-result filter, not a general read-authorization boundary: direct single-record `get(id)` does not consult it — use [`allowRead`](../resources/resource-api.md) for request-level authorization. The same synchronous/side-effect-free constraints as `vectorFilter` apply. + +### Tuning Filtered Traversal + +Filtered traversal is bounded by a visit budget of `ef * filterExpansion` nodes (`filterExpansion` defaults to 24). If the budget is exhausted before the result list fills — which happens when the filter matches only a tiny fraction of records — the search returns the matches found so far rather than erroring. Both knobs can be set per query: + +```javascript +let results = Document.search( + { + sort: { attribute: 'textEmbeddings', target: searchVector, ef: 200, filterExpansion: 40 }, + vectorFilter: (record) => record.category === 'rare', + limit: 10, + }, + context +); +``` + +Raise `filterExpansion` (or `ef`) to trade latency for recall under selective function predicates. Condition-based filters rarely need tuning: very selective conditions are automatically diverted to the exact-scan strategy instead of graph traversal. + ### Filtering by Distance Threshold To return only records whose distance to a target vector is below a threshold, place `target` directly on the condition (alongside `comparator` and `value`). This returns matches within the threshold without using `sort`: @@ -565,6 +619,7 @@ let results = Document.search({ | `mL` | computed from `M` | Normalization factor for level generation | | `efConstructionSearch` | auto-scaled | Max nodes explored during search. When unset, auto-scales with index size (see above); setting it (or `efConstruction`, which seeds it) fixes the budget | | `quantization` | — | `"int8"` stores vectors quantized to int8 (added in v5.1.0, see below) | +| `filterExpansion` | `24` | Visit-budget multiplier for filtered (predicate-aware) search: a filtered query visits at most `ef * filterExpansion` nodes (added in v5.2.0, see above) | Example with custom parameters: diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 365ddf38..f83bdb32 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -8,6 +8,12 @@ title: '5.2' All patch release notes for 5.2.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.2&expanded=true). +## Querying + +### Filtered Vector Search (Predicate-Aware HNSW Traversal) + +Vector searches combined with filters now evaluate the filter during HNSW graph traversal, so the query keeps exploring until it has enough matching nearest neighbors instead of post-filtering a fixed candidate set (which under-filled results under selective filters). Filters can come from query conditions, a JS-API `vectorFilter` function, or a new record-level `allowReadRecord(user, record)` access-control hook — with the RBAC hook, a restricted user receives the k nearest records they are allowed to see. Very selective conditions automatically use an exact scan instead of graph traversal, and a `filterExpansion` visit budget bounds traversal cost. See [Vector Indexing](/reference/v5/database/schema#vector-indexing). + ## Configuration ### Replicated `set_configuration` From 83234d70524d9094cf0308d0654b198e52ba5ebc Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 12 Jul 2026 10:59:00 -0600 Subject: [PATCH 2/5] docs review: state boolean return contract for vectorFilter and allowReadRecord Co-Authored-By: Claude Fable 5 --- reference/database/schema.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/database/schema.md b/reference/database/schema.md index 6dfa2517..f641431a 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -526,13 +526,13 @@ let results = Document.search( ); ``` -`vectorFilter` is available from the JavaScript API only (it cannot be expressed in a REST query string). The function must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). Records passed to it are frozen. +`vectorFilter` is available from the JavaScript API only (it cannot be expressed in a REST query string). The function receives the candidate record and must return a boolean — `true` to include the record in results, `false` to exclude it (it still routes traversal either way). It must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). Records passed to it are frozen. ### Record-Level Access Control in Vector Search -A resource class can define a static `allowReadRecord(user, record)` check that filters query results per record. For vector queries it participates in the traversal, so a restricted user receives the k nearest records _they are allowed to see_ rather than "nearest k, minus redacted" (which under-fills results and reveals that nearby restricted records exist): +A resource class can define a static `allowReadRecord(user, record)` check that filters query results per record, returning a boolean — `true` if the user may see the record, `false` to withhold it. For vector queries it participates in the traversal, so a restricted user receives the k nearest records _they are allowed to see_ rather than "nearest k, minus redacted" (which under-fills results and reveals that nearby restricted records exist): ```javascript export class Reports extends tables.Reports { From 9bf4e256c4f140b2d6967e5b635b2863590dc297 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 13 Jul 2026 23:03:25 -0600 Subject: [PATCH 3/5] Rework record-level access control docs: record-scoped allowRead override replaces allowReadRecord Matches the unified model (record-scoped allowRead, #1422 gap 2) that supersedes the static allowReadRecord hook before 5.2 ships. Co-Authored-By: Claude Fable 5 --- reference/database/schema.md | 17 ++++++++++++----- release-notes/v5-lincoln/5.2.md | 2 +- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/reference/database/schema.md b/reference/database/schema.md index f641431a..beb1d641 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -528,21 +528,28 @@ let results = Document.search( `vectorFilter` is available from the JavaScript API only (it cannot be expressed in a REST query string). The function receives the candidate record and must return a boolean — `true` to include the record in results, `false` to exclude it (it still routes traversal either way). It must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). Records passed to it are frozen. -### Record-Level Access Control in Vector Search +### Record-Level Access Control (Record-Scoped `allowRead`) -A resource class can define a static `allowReadRecord(user, record)` check that filters query results per record, returning a boolean — `true` if the user may see the record, `false` to withhold it. For vector queries it participates in the traversal, so a restricted user receives the k nearest records _they are allowed to see_ rather than "nearest k, minus redacted" (which under-fills results and reveals that nearby restricted records exist): +Overriding `allowRead(user, target, context)` on a table resource makes it a **record-scoped** check: during query execution it is evaluated once per record with `this` bound to the record, so row-level logic reads naturally from `this`. For vector queries the check participates in the graph traversal, so a restricted user receives the k nearest records _they are allowed to see_ rather than "nearest k, minus redacted" (which under-fills results and reveals that nearby restricted records exist): ```javascript export class Reports extends tables.Reports { - static allowReadRecord(user, record) { - return record.ownerId === user.id || user.role?.permissions?.super_user; + allowRead(user, target, context) { + // super = the table/RBAC permission check (safe at any scope) + return super.allowRead(user, target, context) && (user.role.permission.super_user || this.ownerId === user.id); } } ``` -`allowReadRecord` applies to all query results from `search()` (vector or not; on caching tables it is enforced against the record actually returned, after any source revalidation). It is a query-result filter, not a general read-authorization boundary: direct single-record `get(id)` does not consult it — use [`allowRead`](../resources/resource-api.md) for request-level authorization. The same synchronous/side-effect-free constraints as `vectorFilter` apply. +How the one definition applies at each scope (when permission checking is active, e.g. any external request): + +- **Collection queries** (REST collection `GET`, `search()`, including vector sorts) — evaluated per record; rows failing the check are filtered out of results. The default (non-overridden) `allowRead` is a table-level RBAC check and continues to run once at request entry with no per-record cost. +- **Single-record `get(id)`** — evaluated at request entry with the record loaded (attribute reads like `this.ownerId` resolve against the record); a denied record returns a 403. +- **Subscriptions** — a record-scoped override currently fails closed at subscribe time (per-event delivery checks are a planned follow-up). + +Constraints: the check must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). `this` is the frozen record during per-record evaluation. A thrown exception denies that record (fail closed). On caching tables the check is enforced against the record actually returned, after any source revalidation. ### Tuning Filtered Traversal diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index f83bdb32..8cea990a 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -12,7 +12,7 @@ All patch release notes for 5.2.x are available on the [releases page](https://g ### Filtered Vector Search (Predicate-Aware HNSW Traversal) -Vector searches combined with filters now evaluate the filter during HNSW graph traversal, so the query keeps exploring until it has enough matching nearest neighbors instead of post-filtering a fixed candidate set (which under-filled results under selective filters). Filters can come from query conditions, a JS-API `vectorFilter` function, or a new record-level `allowReadRecord(user, record)` access-control hook — with the RBAC hook, a restricted user receives the k nearest records they are allowed to see. Very selective conditions automatically use an exact scan instead of graph traversal, and a `filterExpansion` visit budget bounds traversal cost. See [Vector Indexing](/reference/v5/database/schema#vector-indexing). +Vector searches combined with filters now evaluate the filter during HNSW graph traversal, so the query keeps exploring until it has enough matching nearest neighbors instead of post-filtering a fixed candidate set (which under-filled results under selective filters). Filters can come from query conditions, a JS-API `vectorFilter` function, or a record-scoped `allowRead` override — overriding `allowRead` on a table now makes it a row-level access check, evaluated per record with `this` bound to the record (closing the gap where a collection scan could return rows a single-record GET would deny). With it, a restricted user's vector search returns the k nearest records they are allowed to see. Very selective conditions automatically use an exact scan instead of graph traversal, and a `filterExpansion` visit budget bounds traversal cost. See [Vector Indexing](/reference/v5/database/schema#vector-indexing). ## Configuration From e1923bab9eacc2c7409a94691b7ebf5f192e70d9 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 14 Jul 2026 05:38:17 -0600 Subject: [PATCH 4/5] docs: subscriptions now filter delivery per event through record-scoped allowRead (#1419) Co-Authored-By: Claude Fable 5 --- reference/database/schema.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/database/schema.md b/reference/database/schema.md index beb1d641..4f401ef8 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -547,7 +547,7 @@ How the one definition applies at each scope (when permission checking is active - **Collection queries** (REST collection `GET`, `search()`, including vector sorts) — evaluated per record; rows failing the check are filtered out of results. The default (non-overridden) `allowRead` is a table-level RBAC check and continues to run once at request entry with no per-record cost. - **Single-record `get(id)`** — evaluated at request entry with the record loaded (attribute reads like `this.ownerId` resolve against the record); a denied record returns a 403. -- **Subscriptions** — a record-scoped override currently fails closed at subscribe time (per-event delivery checks are a planned follow-up). +- **Subscriptions** (SSE, WebSocket, MQTT) — delivery is filtered per event: a subscriber receives only the row-change events for records the check permits. Delete tombstones and published message payloads do not carry the full record, so an override keyed on row fields will deny those event types. Constraints: the check must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). `this` is the frozen record during per-record evaluation. A thrown exception denies that record (fail closed). On caching tables the check is enforced against the record actually returned, after any source revalidation. From 950c7729c748b266eab6d43a155e1ea8f57f5a42 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 14 Jul 2026 06:36:31 -0600 Subject: [PATCH 5/5] docs: collection-aware record-scoped allowRead example (super composition + subscribe re-auth) Co-Authored-By: Claude Fable 5 --- reference/database/schema.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/reference/database/schema.md b/reference/database/schema.md index 4f401ef8..7ebaa1ea 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -537,8 +537,15 @@ Overriding `allowRead(user, target, context)` on a table resource makes it a **r ```javascript export class Reports extends tables.Reports { allowRead(user, target, context) { - // super = the table/RBAC permission check (safe at any scope) - return super.allowRead(user, target, context) && (user.role.permission.super_user || this.ownerId === user.id); + // Compose the table/RBAC grant first, so losing the role's read denies (at request entry and + // when a live subscription is re-authorized). super.allowRead is safe to call at any scope. + if (!super.allowRead(user, target, context)) return false; + if (user.role.permission.super_user) return true; + // Collection scope — a whole-table subscribe or the subscription re-auth check — has no record + // loaded (`this.ownerId` is undefined). Return true to open the connection; rows are filtered + // per record during delivery / query execution. + if (this.ownerId == null) return true; + return this.ownerId === user.id; // per record } } ``` @@ -547,7 +554,7 @@ How the one definition applies at each scope (when permission checking is active - **Collection queries** (REST collection `GET`, `search()`, including vector sorts) — evaluated per record; rows failing the check are filtered out of results. The default (non-overridden) `allowRead` is a table-level RBAC check and continues to run once at request entry with no per-record cost. - **Single-record `get(id)`** — evaluated at request entry with the record loaded (attribute reads like `this.ownerId` resolve against the record); a denied record returns a 403. -- **Subscriptions** (SSE, WebSocket, MQTT) — delivery is filtered per event: a subscriber receives only the row-change events for records the check permits. Delete tombstones and published message payloads do not carry the full record, so an override keyed on row fields will deny those event types. +- **Subscriptions** (SSE, WebSocket, MQTT) — the entry check grants the connection at subscribe time (evaluated at collection scope — return the base grant to open), then delivery is filtered per event so a subscriber receives only the row-change events for records the check permits. A live subscription is periodically re-authorized by re-running this same `allowRead` against the current user, so revoking the role's read (or, for a connection-level override, its grant) tears the subscription down. Delete tombstones and published message payloads do not carry the full record, so an override keyed on row fields will deny those event types. Constraints: the check must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). `this` is the frozen record during per-record evaluation. A thrown exception denies that record (fail closed). On caching tables the check is enforced against the record actually returned, after any source revalidation.