docs(sql): 5.2 SQL engine performance guidance#588
Conversation
Document the new Resource-API SQL engine landing in 5.2 (harper #1285): which queries run index-served/streaming vs. which fall back to the legacy full-scan path, and the auto-fallback contract. Replace the blanket "SQL not recommended" warning with accurate, nuanced guidance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the SQL documentation for Harper 5.2, introducing the new SQL engine built on indexed storage and the Resource API. It replaces the previous warning against SQL use in production with an informative block and adds a comprehensive 'Query Performance' section detailing index-served queries, full-scan fallbacks, and practical optimization guidance. There are no review comments, so no further feedback is provided.
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-588 This preview will update automatically when you push new commits. |
…full scan Table.search executes a top-level OR as a union of per-condition index searches (resources/search.ts), and the new engine pushes an OR group down when every branch is index-driving (whereToConditions conditionUsesIndex). So `a = 1 OR b = 2` with both attributes indexed is index-served; only an OR with an un-indexed / non-index-comparator branch falls back to a scan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Follow-up commit (14d4523) corrects the |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-588 This preview will update automatically when you push new commits. |
| :::warning | ||
| SQL querying is not recommended for production use or on large tables. SQL queries often do not utilize indexes and are not optimized for performance. Use the [REST interface](../rest/overview.md) for production data access — it provides a more stable, secure, and performant interface. SQL is intended for ad-hoc data investigation and administrative queries. | ||
| :::info Harper 5.2 introduces a new SQL engine | ||
| As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the same results but with the older full-scan cost. See [Query Performance](#query-performance) for how to keep your queries on the fast path. |
There was a problem hiding this comment.
This is cool, how do users get observability into the ability to plan?
|
|
||
| ## Query Performance | ||
|
|
||
| Harper 5.2's SQL engine maps SQL clauses onto Harper's native indexed storage. When a query can be planned against an index, it runs as a streaming, index-driven operation with memory bounded by the size of the result — not the size of the table. When it can't, the engine falls back to the legacy path, which fetches candidate rows and evaluates the query in memory, with cost and memory growing with the number of rows scanned. |
There was a problem hiding this comment.
There should be some form of larger callout here that the fallback incurs full table scans which is problematic for large tables.
|
|
||
| These can't be served from an index. They still return correct results, but the engine falls back to the legacy full-scan path, so cost and memory grow with table size: | ||
|
|
||
| - **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index. |
There was a problem hiding this comment.
What if there is another indexed filter in an AND?
|
|
||
| ### Practical guidance | ||
|
|
||
| - Index every attribute you filter, sort, or join on. |
There was a problem hiding this comment.
Is it worth calling out performance and storage implications of over indexing?
Adds a 'Behavior changes from the legacy engine' subsection covering the standard-SQL corrections the new engine applies to queries it plans (empty-set aggregates -> NULL, string MIN/MAX, NULL!=NULL join keys, explicit-null expression results), and scopes the fallback-contract 'results don't change' promise to genuine fallbacks. Adds TOTAL to the reserved-word list and notes that reserved words used as AS aliases must be quoted (e.g. AS `total`). Sourced from the sql-engine-v2 QA differential wave (findings D-217, F-143). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-588 This preview will update automatically when you push new commits. |
…equirement; OFFSET behavior change QA verification against the engine (sql-engine-v2 @ 8a46e0302) found three inaccuracies: - The fallback log line is 'SQL engine v2 fallback:' (router.ts), not 'sql-engine v2 fallback:' — the documented grep string matched nothing. - ORDER BY / GROUP BY on an indexed attribute run on the legacy path unless an index-driving WHERE condition is present (validateScannable requires a scan driver) — state the requirement instead of promising them unconditionally. - Behavior changes: OFFSET past the end of a result now correctly returns no rows (legacy could still return rows). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-588 This preview will update automatically when you push new commits. |
2 participants
Adds guidance to the SQL operations-api reference for the new Resource-API SQL engine landing in 5.2 (harper #1285 — New SQL engine on the Resource API), per Kris's request to document which SQL is well supported / runs efficiently and what to avoid.
Changes to
reference/operations-api/sql.md## Query Performancesection:INon indexed attributes, range/BETWEEN, prefixLIKE 'x%', index-orderORDER BYwithLIMIT/OFFSETpushdown,GROUP BYon an indexed attribute,INNER/LEFT JOINon an indexed inner key, projection pushdown.OR, suffix/substringLIKE,!=/NOTnegations, un-drivenORDER BY,SEARCH_JSON; plus a note thatRIGHT/FULL JOIN, no-WHEREUPDATE/DELETE, andCOUNT(DISTINCT)run on the legacy engine.autodefault, results never change (only the performance profile), and thesql-engine v2 fallback:log signal.LIKE, constrain before sort/group/join/SEARCH_JSON, use REST for hot paths.Accuracy notes
LIKE/BETWEEN, sort-from-index, validate-scannable) and the phase-5 cutover parity work in #1285.UNION, sub-SELECT, andINSERT … SELECTare unsupported by both engines, so they are called out as "not supported" rather than "slow fallback."RIGHT/FULL JOINfallback claim against source:normalizer.tsthrowsEngineUnsupportedErrorfor both, and the router catches exactly that underautoto fall back to legacy (which supports them).Note: #1285 is still open and targeted at 5.2; the "As of Harper 5.2" framing lines up with that release. If the merge/version slips, the version reference should move with it.
🤖 Generated with Claude Code