From 23da66d09b83655e37237e7b33793cc485168b33 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 15 Jul 2026 13:33:40 -0600 Subject: [PATCH 1/4] docs(sql): add 5.2 SQL engine performance guidance Document the new Resource-API SQL engine landing in 5.2 (harper #1285): which queries run index-served/streaming vs. which fall back to the legacy full-scan path, and the auto-fallback contract. Replace the blanket "SQL not recommended" warning with accurate, nuanced guidance. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/operations-api/sql.md | 56 +++++++++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 2 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index d27fec67..b1dd6ad1 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -11,8 +11,10 @@ title: SQL -:::warning -SQL querying is not recommended for production use or on large tables. SQL queries often do not utilize indexes and are not optimized for performance. Use the [REST interface](../rest/overview.md) for production data access — it provides a more stable, secure, and performant interface. SQL is intended for ad-hoc data investigation and administrative queries. +:::info Harper 5.2 introduces a new SQL engine +As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the same results but with the older full-scan cost. See [Query Performance](#query-performance) for how to keep your queries on the fast path. + +For your most performance-critical production read paths, the [REST interface](../rest/overview.md) remains the recommended choice — it provides a stable, cacheable, index-driven contract. SQL is well suited to ad-hoc investigation, administrative queries, joins, and reporting. ::: Harper includes a SQL interface supporting SELECT, INSERT, UPDATE, and DELETE operations. Tables are referenced using `database.table` notation (e.g., `dev.dog`). @@ -97,6 +99,56 @@ ORDER BY d.dog_name --- +## Query Performance + +Harper 5.2's SQL engine maps SQL clauses onto Harper's native indexed storage. When a query can be planned against an index, it runs as a streaming, index-driven operation with memory bounded by the size of the result — not the size of the table. When it can't, the engine falls back to the legacy path, which fetches candidate rows and evaluates the query in memory, with cost and memory growing with the number of rows scanned. + +On any non-trivial table the difference between these two paths is large. The rules below keep your queries on the fast path. + +### Index your predicates, sorts, and join keys + +The single most important factor is whether the attributes you filter, sort, and join on are indexed. An equality or range filter on an indexed attribute becomes an index lookup; the same filter on an un-indexed attribute forces a full scan. Define indexes on the attributes your queries actually constrain. + +### Queries that run efficiently (index-served) + +These map directly to indexed storage operations: + +- **Primary-key lookups** — `WHERE id = ...`, `WHERE id IN (...)`. +- **Equality and `IN` on an indexed attribute** — `WHERE status = 'active'`, `WHERE breed_id IN (1, 2, 3)`. +- **Range predicates on an indexed attribute** — `<`, `<=`, `>`, `>=`, and `BETWEEN` (compiled to a bounded index range). +- **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. +- **`ORDER BY` on an indexed attribute** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. +- **`GROUP BY` on an indexed attribute** — aggregates stream group-by-group with O(1) memory per group. +- **`INNER`/`LEFT JOIN` on an indexed join key** — executed as an index-nested-loop join (stream the outer table, probe the inner by index). Keep the join key indexed on the inner (probed) table. +- **Column projections** — selecting a subset of columns pushes down so only those attributes are read. + +### Queries that fall back to a full scan (avoid on large tables) + +These can't be served from an index. They still return correct results, but the engine falls back to the legacy full-scan path, so cost and memory grow with table size: + +- **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index. +- **`OR` across different attributes** — e.g. `WHERE a = 1 OR b = 2`. (An `IN` list on a _single_ indexed attribute is fine.) +- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. +- **`!=` / `<>` and `NOT` negations** — matching "everything except" a value can't use an index. +- **`ORDER BY` with no indexed filter to drive it** — a table-wide ordered scan on an un-indexed sort key. +- **`SEARCH_JSON`** — evaluates JSONata over each candidate row's nested JSON; it is not index-backed. Narrow the candidate set with an indexed `WHERE` condition first. +- **`RIGHT`/`FULL OUTER JOIN`, no-`WHERE` `UPDATE`/`DELETE`, `COUNT(DISTINCT ...)`** — the new engine doesn't plan these yet, so they run on the legacy engine (correct results, legacy cost). + +Note that `UNION`, sub-`SELECT`, and `INSERT … SELECT` are not supported by either engine — see the [Features Matrix](#features-matrix). + +### The fallback contract + +The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't, so **no query changes its results** — only its performance profile. Fallbacks are logged (`sql-engine v2 fallback: ...`) with the reason and the offending column, which is a useful signal for finding queries worth an added index or a reshape. + +### Practical guidance + +- Index every attribute you filter, sort, or join on. +- Prefer prefix `LIKE 'x%'` over `LIKE '%x%'`; back a substring search with a separate indexed predicate, or model it as a dedicated indexed attribute. +- Constrain the row set with an indexed `WHERE` before leaning on `ORDER BY`, `GROUP BY`, joins, or `SEARCH_JSON`. +- Reach for the [REST interface](../rest/overview.md) on your hottest production read paths for a cacheable, index-driven contract; keep SQL for ad-hoc investigation, joins, and reporting. + +--- + ## Features Matrix | INSERT | | From 14d45235c835fa62ad84fa2d265794ee96688349 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 15 Jul 2026 14:13:30 -0600 Subject: [PATCH 2/4] =?UTF-8?q?docs(sql):=20correct=20OR=20handling=20?= =?UTF-8?q?=E2=80=94=20indexed=20OR=20is=20an=20index=20union,=20not=20a?= =?UTF-8?q?=20full=20scan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Table.search executes a top-level OR as a union of per-condition index searches (resources/search.ts), and the new engine pushes an OR group down when every branch is index-driving (whereToConditions conditionUsesIndex). So `a = 1 OR b = 2` with both attributes indexed is index-served; only an OR with an un-indexed / non-index-comparator branch falls back to a scan. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/operations-api/sql.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index b1dd6ad1..4ed66c8c 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -116,6 +116,7 @@ These map directly to indexed storage operations: - **Primary-key lookups** — `WHERE id = ...`, `WHERE id IN (...)`. - **Equality and `IN` on an indexed attribute** — `WHERE status = 'active'`, `WHERE breed_id IN (1, 2, 3)`. - **Range predicates on an indexed attribute** — `<`, `<=`, `>`, `>=`, and `BETWEEN` (compiled to a bounded index range). +- **`OR` of indexed predicates** — `WHERE a = 1 OR b = 2` is served as a union of index scans, provided **every** branch is an index-driving predicate on an indexed attribute. - **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. - **`ORDER BY` on an indexed attribute** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. - **`GROUP BY` on an indexed attribute** — aggregates stream group-by-group with O(1) memory per group. @@ -127,8 +128,8 @@ These map directly to indexed storage operations: These can't be served from an index. They still return correct results, but the engine falls back to the legacy full-scan path, so cost and memory grow with table size: - **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index. -- **`OR` across different attributes** — e.g. `WHERE a = 1 OR b = 2`. (An `IN` list on a _single_ indexed attribute is fine.) -- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. +- **`OR` where a branch isn't index-driving** — e.g. `WHERE a = 1 OR b = 2` when `b` is un-indexed, or `WHERE a = 1 OR name LIKE '%x%'`. A single un-indexed (or non-index-comparator) branch taints the whole disjunction and forces a scan. When **every** branch is an indexed, index-driving predicate, the `OR` stays on the fast path (see above). +- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined (via `AND`) with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. - **`!=` / `<>` and `NOT` negations** — matching "everything except" a value can't use an index. - **`ORDER BY` with no indexed filter to drive it** — a table-wide ordered scan on an un-indexed sort key. - **`SEARCH_JSON`** — evaluates JSONata over each candidate row's nested JSON; it is not index-backed. Narrow the candidate set with an indexed `WHERE` condition first. From a19e1a39d499f7cc6087e42e38430c401caeca07 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 15 Jul 2026 15:28:48 -0600 Subject: [PATCH 3/4] docs(sql): document 5.2 engine behavior changes + reserved-word aliases Adds a 'Behavior changes from the legacy engine' subsection covering the standard-SQL corrections the new engine applies to queries it plans (empty-set aggregates -> NULL, string MIN/MAX, NULL!=NULL join keys, explicit-null expression results), and scopes the fallback-contract 'results don't change' promise to genuine fallbacks. Adds TOTAL to the reserved-word list and notes that reserved words used as AS aliases must be quoted (e.g. AS `total`). Sourced from the sql-engine-v2 QA differential wave (findings D-217, F-143). Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/operations-api/sql.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 4ed66c8c..312bfc2f 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -139,7 +139,18 @@ Note that `UNION`, sub-`SELECT`, and `INSERT … SELECT` are not supported by ei ### The fallback contract -The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't, so **no query changes its results** — only its performance profile. Fallbacks are logged (`sql-engine v2 fallback: ...`) with the reason and the offending column, which is a useful signal for finding queries worth an added index or a reshape. +The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would — **its results don't change, only its performance profile.** Fallbacks are logged (`sql-engine v2 fallback: ...`) with the reason and the offending column, which is a useful signal for finding queries worth an added index or a reshape. (For queries the new engine _does_ plan, it corrects a few long-standing legacy quirks — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) + +### Behavior changes from the legacy engine + +For queries the new engine plans (the default under `auto`), a handful of non-standard legacy behaviors are corrected, so you may see different — standard-SQL-conformant — results after upgrading: + +- **Aggregates over an empty result set** return `NULL` for `SUM`, `AVG`, `MIN`, and `MAX` (and `0` for `COUNT`), per the SQL standard. The legacy engine returned `0` for `SUM` and omitted `MIN`/`MAX` from the response entirely. +- **`MIN`/`MAX` over text columns** return the lexicographically smallest/largest value. The legacy engine produced no result for string `MIN`/`MAX`. +- **`NULL` never equals `NULL` in a join key.** Rows whose join key is `NULL` on either side no longer match each other (three-valued logic). The legacy engine incorrectly joined `NULL` to `NULL`. +- **A `NULL`-valued expression result is returned as an explicit `null`** rather than omitted from the row. A computed column or `COALESCE` that evaluates to `NULL` — or `UPPER()` — now yields `null` (the legacy engine dropped the key, and `UPPER` of a `NULL` returned the literal string `"NULL"`). + +These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior. ### Practical guidance @@ -383,16 +394,17 @@ Geospatial data must be stored using the [GeoJSON standard](https://geojson.org/ ## Reserved Words -If a database, table, or attribute name conflicts with a reserved word, wrap it in backticks or brackets: +If a database, table, attribute, or column-alias name conflicts with a reserved word, wrap it in backticks or brackets. This applies to `AS` aliases too: an unquoted reserved word such as `AS total` fails to parse — quote it, as in the last example below. ```sql SELECT * FROM data.`ASSERT` SELECT * FROM data.[ASSERT] +SELECT SUM(qty) AS `total` FROM data.dog ```
Full reserved word list -ABSOLUTE, ACTION, ADD, AGGR, ALL, ALTER, AND, ANTI, ANY, APPLY, ARRAY, AS, ASSERT, ASC, ATTACH, AUTOINCREMENT, AUTO_INCREMENT, AVG, BEGIN, BETWEEN, BREAK, BY, CALL, CASE, CAST, CHECK, CLASS, CLOSE, COLLATE, COLUMN, COLUMNS, COMMIT, CONSTRAINT, CONTENT, CONTINUE, CONVERT, CORRESPONDING, COUNT, CREATE, CROSS, CUBE, CURRENT_TIMESTAMP, CURSOR, DATABASE, DECLARE, DEFAULT, DELETE, DELETED, DESC, DETACH, DISTINCT, DOUBLEPRECISION, DROP, ECHO, EDGE, END, ENUM, ELSE, EXCEPT, EXISTS, EXPLAIN, FALSE, FETCH, FIRST, FOREIGN, FROM, GO, GRAPH, GROUP, GROUPING, HAVING, HDB_HASH, HELP, IF, IDENTITY, IS, IN, INDEX, INNER, INSERT, INSERTED, INTERSECT, INTO, JOIN, KEY, LAST, LET, LEFT, LIKE, LIMIT, LOOP, MATCHED, MATRIX, MAX, MERGE, MIN, MINUS, MODIFY, NATURAL, NEXT, NEW, NOCASE, NO, NOT, NULL, OFF, ON, ONLY, OFFSET, OPEN, OPTION, OR, ORDER, OUTER, OVER, PATH, PARTITION, PERCENT, PLAN, PRIMARY, PRINT, PRIOR, QUERY, READ, RECORDSET, REDUCE, REFERENCES, RELATIVE, REPLACE, REMOVE, RENAME, REQUIRE, RESTORE, RETURN, RETURNS, RIGHT, ROLLBACK, ROLLUP, ROW, SCHEMA, SCHEMAS, SEARCH, SELECT, SEMI, SET, SETS, SHOW, SOME, SOURCE, STRATEGY, STORE, SYSTEM, SUM, TABLE, TABLES, TARGET, TEMP, TEMPORARY, TEXTSTRING, THEN, TIMEOUT, TO, TOP, TRAN, TRANSACTION, TRIGGER, TRUE, TRUNCATE, UNION, UNIQUE, UPDATE, USE, USING, VALUE, VERTEX, VIEW, WHEN, WHERE, WHILE, WITH, WORK +ABSOLUTE, ACTION, ADD, AGGR, ALL, ALTER, AND, ANTI, ANY, APPLY, ARRAY, AS, ASSERT, ASC, ATTACH, AUTOINCREMENT, AUTO_INCREMENT, AVG, BEGIN, BETWEEN, BREAK, BY, CALL, CASE, CAST, CHECK, CLASS, CLOSE, COLLATE, COLUMN, COLUMNS, COMMIT, CONSTRAINT, CONTENT, CONTINUE, CONVERT, CORRESPONDING, COUNT, CREATE, CROSS, CUBE, CURRENT_TIMESTAMP, CURSOR, DATABASE, DECLARE, DEFAULT, DELETE, DELETED, DESC, DETACH, DISTINCT, DOUBLEPRECISION, DROP, ECHO, EDGE, END, ENUM, ELSE, EXCEPT, EXISTS, EXPLAIN, FALSE, FETCH, FIRST, FOREIGN, FROM, GO, GRAPH, GROUP, GROUPING, HAVING, HDB_HASH, HELP, IF, IDENTITY, IS, IN, INDEX, INNER, INSERT, INSERTED, INTERSECT, INTO, JOIN, KEY, LAST, LET, LEFT, LIKE, LIMIT, LOOP, MATCHED, MATRIX, MAX, MERGE, MIN, MINUS, MODIFY, NATURAL, NEXT, NEW, NOCASE, NO, NOT, NULL, OFF, ON, ONLY, OFFSET, OPEN, OPTION, OR, ORDER, OUTER, OVER, PATH, PARTITION, PERCENT, PLAN, PRIMARY, PRINT, PRIOR, QUERY, READ, RECORDSET, REDUCE, REFERENCES, RELATIVE, REPLACE, REMOVE, RENAME, REQUIRE, RESTORE, RETURN, RETURNS, RIGHT, ROLLBACK, ROLLUP, ROW, SCHEMA, SCHEMAS, SEARCH, SELECT, SEMI, SET, SETS, SHOW, SOME, SOURCE, STRATEGY, STORE, SYSTEM, SUM, TABLE, TABLES, TARGET, TEMP, TEMPORARY, TEXTSTRING, THEN, TIMEOUT, TO, TOP, TOTAL, TRAN, TRANSACTION, TRIGGER, TRUE, TRUNCATE, UNION, UNIQUE, UPDATE, USE, USING, VALUE, VERTEX, VIEW, WHEN, WHERE, WHILE, WITH, WORK
From e16f88bb1565a4863c5913556ec99ecd42269b6b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 15 Jul 2026 16:48:19 -0600 Subject: [PATCH 4/4] docs(sql): correct fallback-log string; ORDER/GROUP BY WHERE-driver requirement; OFFSET behavior change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit QA verification against the engine (sql-engine-v2 @ 8a46e0302) found three inaccuracies: - The fallback log line is 'SQL engine v2 fallback:' (router.ts), not 'sql-engine v2 fallback:' — the documented grep string matched nothing. - ORDER BY / GROUP BY on an indexed attribute run on the legacy path unless an index-driving WHERE condition is present (validateScannable requires a scan driver) — state the requirement instead of promising them unconditionally. - Behavior changes: OFFSET past the end of a result now correctly returns no rows (legacy could still return rows). Co-Authored-By: Claude Fable 5 --- reference/operations-api/sql.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 312bfc2f..a376e15d 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -118,8 +118,8 @@ These map directly to indexed storage operations: - **Range predicates on an indexed attribute** — `<`, `<=`, `>`, `>=`, and `BETWEEN` (compiled to a bounded index range). - **`OR` of indexed predicates** — `WHERE a = 1 OR b = 2` is served as a union of index scans, provided **every** branch is an index-driving predicate on an indexed attribute. - **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. -- **`ORDER BY` on an indexed attribute** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. -- **`GROUP BY` on an indexed attribute** — aggregates stream group-by-group with O(1) memory per group. +- **`ORDER BY` on an indexed attribute, combined with an indexed `WHERE`** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. Without an index-driving `WHERE` condition, an `ORDER BY` currently runs on the legacy path. +- **`GROUP BY` on an indexed attribute, combined with an indexed `WHERE`** — aggregates stream group-by-group with O(1) memory per group (memory is bounded by the group count; every matching row is still read). Without an index-driving `WHERE` condition, a `GROUP BY` currently runs on the legacy path. - **`INNER`/`LEFT JOIN` on an indexed join key** — executed as an index-nested-loop join (stream the outer table, probe the inner by index). Keep the join key indexed on the inner (probed) table. - **Column projections** — selecting a subset of columns pushes down so only those attributes are read. @@ -139,7 +139,7 @@ Note that `UNION`, sub-`SELECT`, and `INSERT … SELECT` are not supported by ei ### The fallback contract -The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would — **its results don't change, only its performance profile.** Fallbacks are logged (`sql-engine v2 fallback: ...`) with the reason and the offending column, which is a useful signal for finding queries worth an added index or a reshape. (For queries the new engine _does_ plan, it corrects a few long-standing legacy quirks — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) +The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would — **its results don't change, only its performance profile.** Fallbacks are logged (`SQL engine v2 fallback: ...`) with the reason, which is a useful signal for finding queries worth an added index or a reshape. (For queries the new engine _does_ plan, it corrects a few long-standing legacy quirks — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) ### Behavior changes from the legacy engine @@ -148,6 +148,7 @@ For queries the new engine plans (the default under `auto`), a handful of non-st - **Aggregates over an empty result set** return `NULL` for `SUM`, `AVG`, `MIN`, and `MAX` (and `0` for `COUNT`), per the SQL standard. The legacy engine returned `0` for `SUM` and omitted `MIN`/`MAX` from the response entirely. - **`MIN`/`MAX` over text columns** return the lexicographically smallest/largest value. The legacy engine produced no result for string `MIN`/`MAX`. - **`NULL` never equals `NULL` in a join key.** Rows whose join key is `NULL` on either side no longer match each other (three-valued logic). The legacy engine incorrectly joined `NULL` to `NULL`. +- **`OFFSET` past the end of a result returns no rows.** The legacy engine could still return rows when the `OFFSET` exceeded the number of matches. - **A `NULL`-valued expression result is returned as an explicit `null`** rather than omitted from the row. A computed column or `COALESCE` that evaluates to `NULL` — or `UPPER()` — now yields `null` (the legacy engine dropped the key, and `UPPER` of a `NULL` returned the literal string `"NULL"`). These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior.