fix(catalog): describe-first table resolution for REST namespaces

[hellower]

Problem

#220 added a lazy-discovery guard to LanceRestNamespaceDefaultGenerator::CreateDefaultEntry: before resolving a name, it checks membership in GetDefaultEntries() (a list_tables round trip). The guard's purpose is sound — DuckDB probes the active catalog for system names (e.g. duckdb_tables when SHOW TABLES runs under USE <lance_catalog>), and those probes must fall through to the system catalog instead of aborting the statement. But making a list round trip the gate for every point lookup has three defects:

1. Stale credentials after secret rotation. The membership list runs with the ATTACH-time captured bearer_token/api_key (GetDefaultEntries() uses the member fields), before ResolveLanceNamespaceAuth re-resolves DuckDB secrets. After a secret rotation, point lookups of not-yet-materialized entries fail with stale credentials even though the describe/open path below — which does use freshly resolved credentials — would succeed.

2. Already-qualified references are rejected. With a non-empty namespace id, GetDefaultEntries() strips the namespace prefix from listed names, but the guard compares the raw entry name against the stripped list: a qualified reference (ns$tbl) is rejected by the guard before the candidate logic below (which deliberately describes already-prefixed names as-is) ever runs.

3. List permission becomes a prerequisite for point lookups. Credentials that allow describe/open of a specific table but deny list_tables, or tables intentionally omitted from listings, are misreported as not found.

Fix: describe-first resolution

Drop the up-front membership gate; resolve auth first; describe the candidates with three-state outcomes — Found / NotFound / Error — where NotFound is a new typed signal instead of a guess:

• Rust: new ErrorCode::NamespaceTableNotFound, reported by describe_table_with_schema_inner via a classifier that only treats spec-conformant not-found as typed: lance_core::Error::NotFound, or a NamespaceError::TableNotFound / NamespaceNotFound downcast out of Error::Namespace. The REST client materializes those variants from the numeric code field of spec error bodies (NamespaceError::from_code); non-spec error bodies deserialize to other variants and deliberately remain generic describe errors.

• C++ candidates (canonical id = the namespace-qualified form, probed FIRST; the bare name is only an alias for server dialects that store unprefixed ids). On a hierarchical server a one-segment id addresses the ROOT namespace, so the alias is double-guarded:

  • once the canonical id proves existence (describe Found or open succeeded, schema convertibility aside) the alias is never consulted — a same-named root table must not shadow the attached namespace's table;
  • an alias describe/open success only counts after a memoized membership listing confirms the name belongs to the attached namespace — checked lazily after the success, so misses (e.g. system-name probes) never pay a list round trip;
  • alias NotFound/Error outcomes are never authoritative (arity noise from strict multi-level servers must not force the membership fallback or an IOException under list-denied credentials).

• C++ precedence on the canonical outcome:

  1. any candidate found but unconvertible (describe or open succeeded, schema conversion failed) → empty entry: the table provably exists, the CatalogSet::CreateDefaultEntries non-null contract for enumeration is preserved, and the real incompatibility surfaces at use time;
  2. canonical typed NotFound → nullptr: soft fall-through to the system catalog — the guard's original purpose;
  3. canonical infra Error → membership fallback with the freshly resolved credentials (same memoized listing as the alias gate): unlisted → nullptr (that server's not-found dialect), listed → empty entry, and if the list itself fails the canonical candidate's describe error surfaces as an IOException — a possibly-existing table is never masqueraded as "not found".

  The thread-local FFI error is consumed on every path — including the slow fallback's failed-open paths — so it cannot leak into an unrelated LanceFormatErrorSuffix() later.

Trade-off (from review): on server dialects that store unprefixed ids, alias resolution now requires list permission to confirm membership; the canonical path still needs none. This is the price of not letting a hierarchical server's root namespace answer for an attached child namespace.

Also adds rust/ffi/namespace.rs to RUST_FFI_DEPENDS in CMakeLists.txt — the hand-maintained dependency list predates the file, so incremental builds would not otherwise pick up this change.

Error code numbering

ErrorCode 55 is intentionally left unused: the in-flight #225 claims 55 for its own new code, and skipping it lets the two changes merge in either order without a value collision in the C++ mirror constant (the enum already tolerates a gap at 33).

Tests

• Rust: table-driven classifier test with a minimal in-process listener — a reachable server answering a spec-shaped TableNotFound error body (numeric code 4) must map to the typed FFI code, while an unreachable endpoint must stay a generic describe error.
• C++ REST behavior remains covered by the env-gated LANCE_TEST_NAMESPACE tests; the sqllogictest suite is unchanged from baseline (56/57 pass locally, the one failure — scan_limit_through_filter.test — is pre-existing on the base branch).

Context

These defects predate and are independent of #225 (found while addressing its review feedback on a downstream port). Refs #220.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ea923158c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[hellower]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: fa003b9619

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".