diff --git a/reference/database/schema.md b/reference/database/schema.md index 5b342f06..7ebaa1ea 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -507,6 +507,74 @@ let results = Document.search({ }); ``` + — Conditions combined with a vector sort are evaluated _during_ graph traversal (predicate-aware search): the search keeps exploring until it has enough _matching_ nearest neighbors, instead of finding the nearest candidates first and then dropping the ones that fail the filter. With a selective filter this is the difference between a full result set and an under-filled one. When a companion condition is very selective, Harper instead computes exact distances over just the records matching that condition, which is both exact and faster than traversing the graph. + +### Filtered Vector Search with a Function Predicate + + + +A `vectorFilter` function on the query participates in the traversal the same way, for predicates that are not expressible as conditions: + +```javascript +let results = Document.search( + { + sort: { attribute: 'textEmbeddings', target: searchVector }, + vectorFilter: (record) => record.tenantId === context.user.tenantId && record.status === 'published', + limit: 10, + }, + context +); +``` + +`vectorFilter` is available from the JavaScript API only (it cannot be expressed in a REST query string). The function receives the candidate record and must return a boolean — `true` to include the record in results, `false` to exclude it (it still routes traversal either way). It must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). Records passed to it are frozen. + +### Record-Level Access Control (Record-Scoped `allowRead`) + + + +Overriding `allowRead(user, target, context)` on a table resource makes it a **record-scoped** check: during query execution it is evaluated once per record with `this` bound to the record, so row-level logic reads naturally from `this`. For vector queries the check participates in the graph traversal, so a restricted user receives the k nearest records _they are allowed to see_ rather than "nearest k, minus redacted" (which under-fills results and reveals that nearby restricted records exist): + +```javascript +export class Reports extends tables.Reports { + allowRead(user, target, context) { + // Compose the table/RBAC grant first, so losing the role's read denies (at request entry and + // when a live subscription is re-authorized). super.allowRead is safe to call at any scope. + if (!super.allowRead(user, target, context)) return false; + if (user.role.permission.super_user) return true; + // Collection scope — a whole-table subscribe or the subscription re-auth check — has no record + // loaded (`this.ownerId` is undefined). Return true to open the connection; rows are filtered + // per record during delivery / query execution. + if (this.ownerId == null) return true; + return this.ownerId === user.id; // per record + } +} +``` + +How the one definition applies at each scope (when permission checking is active, e.g. any external request): + +- **Collection queries** (REST collection `GET`, `search()`, including vector sorts) — evaluated per record; rows failing the check are filtered out of results. The default (non-overridden) `allowRead` is a table-level RBAC check and continues to run once at request entry with no per-record cost. +- **Single-record `get(id)`** — evaluated at request entry with the record loaded (attribute reads like `this.ownerId` resolve against the record); a denied record returns a 403. +- **Subscriptions** (SSE, WebSocket, MQTT) — the entry check grants the connection at subscribe time (evaluated at collection scope — return the base grant to open), then delivery is filtered per event so a subscriber receives only the row-change events for records the check permits. A live subscription is periodically re-authorized by re-running this same `allowRead` against the current user, so revoking the role's read (or, for a connection-level override, its grant) tears the subscription down. Delete tombstones and published message payloads do not carry the full record, so an override keyed on row fields will deny those event types. + +Constraints: the check must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). `this` is the frozen record during per-record evaluation. A thrown exception denies that record (fail closed). On caching tables the check is enforced against the record actually returned, after any source revalidation. + +### Tuning Filtered Traversal + +Filtered traversal is bounded by a visit budget of `ef * filterExpansion` nodes (`filterExpansion` defaults to 24). If the budget is exhausted before the result list fills — which happens when the filter matches only a tiny fraction of records — the search returns the matches found so far rather than erroring. Both knobs can be set per query: + +```javascript +let results = Document.search( + { + sort: { attribute: 'textEmbeddings', target: searchVector, ef: 200, filterExpansion: 40 }, + vectorFilter: (record) => record.category === 'rare', + limit: 10, + }, + context +); +``` + +Raise `filterExpansion` (or `ef`) to trade latency for recall under selective function predicates. Condition-based filters rarely need tuning: very selective conditions are automatically diverted to the exact-scan strategy instead of graph traversal. + ### Filtering by Distance Threshold To return only records whose distance to a target vector is below a threshold, place `target` directly on the condition (alongside `comparator` and `value`). This returns matches within the threshold without using `sort`: @@ -565,6 +633,7 @@ let results = Document.search({ | `mL` | computed from `M` | Normalization factor for level generation | | `efConstructionSearch` | auto-scaled | Max nodes explored during search. When unset, auto-scales with index size (see above); setting it (or `efConstruction`, which seeds it) fixes the budget | | `quantization` | — | `"int8"` stores vectors quantized to int8 (added in v5.1.0, see below) | +| `filterExpansion` | `24` | Visit-budget multiplier for filtered (predicate-aware) search: a filtered query visits at most `ef * filterExpansion` nodes (added in v5.2.0, see above) | Example with custom parameters: diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 365ddf38..8cea990a 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -8,6 +8,12 @@ title: '5.2' All patch release notes for 5.2.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.2&expanded=true). +## Querying + +### Filtered Vector Search (Predicate-Aware HNSW Traversal) + +Vector searches combined with filters now evaluate the filter during HNSW graph traversal, so the query keeps exploring until it has enough matching nearest neighbors instead of post-filtering a fixed candidate set (which under-filled results under selective filters). Filters can come from query conditions, a JS-API `vectorFilter` function, or a record-scoped `allowRead` override — overriding `allowRead` on a table now makes it a row-level access check, evaluated per record with `this` bound to the record (closing the gap where a collection scan could return rows a single-record GET would deny). With it, a restricted user's vector search returns the k nearest records they are allowed to see. Very selective conditions automatically use an exact scan instead of graph traversal, and a `filterExpansion` visit budget bounds traversal cost. See [Vector Indexing](/reference/v5/database/schema#vector-indexing). + ## Configuration ### Replicated `set_configuration`