docs: correct the SEARCH table function reference - #786
Conversation
search_columns is required (LanceSearchTableFunctions.search throws when it is empty) but has no positional slot, so the documented positional example always fails and SEARCH in fact requires named arguments. offset is documented but never read by search(); unknown named arguments are silently ignored, so it looks accepted. Execution described the removed bespoke single-partition path; the scan now runs server-side through queryTable only when the namespace supports it, and per-fragment otherwise. Validation claimed Docker coverage while the pytest case is xfail and the JVM case is @disabled.
github-actions
Bot
added
the
documentation
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
| ## Execution | ||
|
|
||
| Spark plans `SEARCH` as a DataSource V2 batch read with one input partition. The partition reader calls the Lance namespace `queryTable` API. With a directory namespace the search runs in the Spark process executing that reader; with a REST namespace the REST server handles the namespace request. | ||
| Spark plans `SEARCH` as a batch read carrying the full-text query as a scan option, wrapped in an optional filter, a projection, `ORDER BY _score DESC`, and `LIMIT k`. The scan then runs one of two ways: a single-partition server-side read through the Lance namespace `queryTable` API when the namespace supports it, or a distributed per-fragment scan for catalog-only namespaces and for reads that target a branch or tag. |
There was a problem hiding this comment.
I haven't see how search can target a branch or tag is this true or should we drop?
There was a problem hiding this comment.
Tag is not reachable, you are right to push on it: version goes through optionalLong, so a tag name never parses, and there is no tag_ identifier suffix to carry one. Branch is reachable, since resolveLanceTable calls loadTable and that matches BRANCH_SUFFIX, so table => '...docs.branch_audit' lands on the branch and shouldNamespaceFtsScan falls back, but nothing tests that path. Dropped the clause in 52baafa and left only the catalog-only condition. Happy to put branch back if you want it documented.
There was a problem hiding this comment.
Two corrections to my last reply. I over-corrected by dropping branch as well as tag, and nothing tests that path was wrong — LanceScanBuilderTest#testBranchFullTextQueryDoesNotUseNamespaceScan (LanceScanBuilderTest.java:374) already asserts shouldNamespaceFtsScan() is false for LanceRef.ofBranch. 4ea4cb3 makes the per-fragment scan the general fallback and lists the three conditions the server-side route needs.
Review feedback: a tag cannot reach SEARCH at all. The version argument goes through optionalLong, so a tag name never parses, and there is no tag_ identifier suffix to carry one. A branch is reachable, since resolveLanceTable calls loadTable and that matches BRANCH_SUFFIX on the identifier, but nothing tests it. Leave the condition users can act on and drop the rest.
lance-gatekeeper
Bot
removed
the
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
I over-corrected: only the tag half of the previous clause was wrong, and dropping branch too made the routing read as if catalog-only namespaces were the sole fallback trigger. State the fallback as the otherwise case and list the three conditions the server-side route needs, matching shouldNamespaceFtsScan: a namespace that implements queryTable, a ref that is not a branch or tag, and no pushed aggregation. Enumerating only the fallback triggers went stale twice.
lance-gatekeeper
Bot
removed
the
K-changes
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The latest revision fixes the remaining routing description: the per-fragment scan is now the general fallback, while the three prerequisites for the server-side path match the implementation. The corrected SEARCH argument and validation guidance remains accurate, and the strict documentation build passes.
lance-gatekeeper
Bot
added
the
K-approved
2 participants
Summary
search_columnsis required with no positional slot, soSEARCHneeds named args (Spark 3.5+)offset, which is never readSEARCHtest passes)Test plan
mkdocs build --strict🤖 Generated with Claude Code