feat(f35): local database browser (SQLite) - #48
Conversation
Local database browser, read side (SQLite only — DuckDB is out of scope this cycle: bundled libduckdb C++ cannot build on the 6 GB MinGW dev machine; documented where sources register in db_browser.rs). safe_query.rs — the SafeQueryEngine foundation F36 builds on: - approved-source registry (canonicalized, user-approved db paths; open documents are the only other source kind) - read-only connection factory with a deny-by-default authorizer: SELECT/Read/read-pragma allowlist only; ATTACH, DDL, DML, PRAGMA writes and load_extension rejected at prepare time - row/byte-capped result materialisation (run_select + QueryLimits) - cancellation via a progress handler polling the job's CancelToken (SQLITE_INTERRUPT maps to AppError::Cancelled) - ceesvee_doc read-only vtab exposing open documents with revision snapshot semantics: snapshot copy up to 200k cells, revision-check- abort beyond (a mid-query edit aborts, never mixes revisions) db_browser.rs — schema browser + document integration: - tables/views with columns, PKs, indexes, FKs, WITHOUT ROWID flag, bounded row-count estimates (sqlite_stat1 or capped scan), previews - indexed read-only table opens: DbTableBacking pages windows by rowid/keyset anchors (one anchor per 4096 rows — the table is never materialised); views fall back to LIMIT/OFFSET - editable import bounded by a sampled memory estimate (force overrides); declared SQL types map onto F31 column schemas - refresh detection: PRAGMA data_version (rows) + schema hash (structure) baselines per session and per open table document; stale window reads fail loudly instead of slicing shifted rows Document gains a Backing::Virtual seam (VirtualRows trait: windowed reads, coalesced scattered visits, refresh probe) reported to the front end as "indexedReadOnly" so every existing read-only affordance applies; JobCtx gains a 'static CancelToken for FFI callbacks. 29 new unit tests: authorizer denial matrix, vtab snapshot/live semantics, rowid-gap + WITHOUT ROWID keyset paging, refresh detection, memory-bounded imports, cancellation releasing the database file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Frontend for the F35 local database browser (SQLite only; DuckDB out of scope this cycle). Wraps the read side and the export write side that the backend stages landed. - types.ts: TS mirrors of the db_browser/db_export DTOs. - tauri.ts: typed wrappers for all 11 db commands. - DatabaseDialog: file picker, schema tree (tables/views with column, PK, FK and index detail plus row estimates), bounded preview pane, and per-object open-read-only / import-editable / refresh actions. Detects external row/schema changes and offers a reload; surfaces the memory bound with an explicit "import anyway". - DbExportDialog: target picker, table + mode (create/append/replace with a replace confirmation), per-column SQL name/type/PK mapping editor, append compatibility display, pre-write conversion-failure list, and a primary-key conflict-policy picker. Runs as a cancellable job and shows the report. - store: dbBrowser + dbExport slices; routes the new dbOpenTable / dbImportTable / dbExport job kinds; opening a .db/.sqlite file routes to the browser. - commandDefs: "Open database…" and "Export to database…". - lib/dbExport.ts: pure mapping/compatibility/gating logic + 18 vitest cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add README Features bullets for the SQLite database browser: opening a database (schema browse, read-only indexed open / editable import, external change detection) under Viewing, and exporting a document to a SQLite table (mapping preview, conversion preflight, one transaction, PK conflict policy) under Editing, with the SQLite-only / DuckDB-out-of-scope fine print. Full gate run green: eslint, prettier --check, tsc, 391 vitest tests, vite build; cargo fmt --check, clippy -D warnings, 737 lib tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- safe_query: bound the document-vtab snapshot copy by accumulated bytes (SNAPSHOT_MAX_BYTES), not cell count alone, so a document with few but very large cells falls back to the live revision-check strategy instead of copying hundreds of MB — enforcing the documented "few MB" bound. - db_export: an empty/null cell mapped onto a NOT NULL target column is now a conversion failure (shown in the preview, aborts the write) rather than a silent SQL NULL that the write would abort on opaquely or "skip" would drop as a fake duplicate. - db_export: the "skip" conflict policy no longer uses INSERT OR IGNORE (which swallows every constraint). It issues a plain INSERT and classifies each violation by SQLite extended code, skipping ONLY PRIMARY KEY / UNIQUE conflicts; NOT NULL / CHECK / FOREIGN KEY violations abort with a clear error, so rows_skipped counts only genuine uniqueness conflicts. Rejected: the "replace has no target-staleness guard" finding. Replace is a user-confirmed unconditional overwrite (confirm_replace); it drops and recreates the table atomically without reading target content, so a concurrent target write produces exactly the coherent state "replace" means and no spec invariant is violated. The suggested PRAGMA data_version guard is also unusable here: data_version is only comparable within one connection, and preview and write use different short-lived connections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebase feat/f35-db onto feat/f34-excel. Resolve additive-union conflicts across lib.rs (invoke_handler + manage), document.rs (Backing::Virtual alongside Backing::Columnar), dto/types/tauri/commandDefs/useStore/App.tsx, CHANGELOG.md and README.md. Regenerate Cargo.lock for the unioned Cargo.toml (arrow/parquet/calamine from F32/F34 + rusqlite for F35), and initialize the new `display_name` field in the F32 `from_columnar` constructor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e126f30c41
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let mut conn = Connection::open_with_flags(path, flags) | ||
| .map_err(|e| AppError::invalid(format!("cannot open database for writing: {e}")))?; | ||
| conn.busy_timeout(BUSY_TIMEOUT)?; |
There was a problem hiding this comment.
When appending/replacing into a table that has FOREIGN KEY constraints, this write connection never runs PRAGMA foreign_keys = ON; SQLite's per-connection default is off, so child rows with missing parents are committed instead of failing and rolling back as the export contract promises. Enable FK enforcement on the export connection before starting the transaction so FK violations are handled like the other constraints.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in abd5221.
Confirmed the gap: run_export opened the write connection and set busy_timeout, but never enabled foreign-key enforcement, and SQLite's per-connection default is foreign_keys = OFF. So an append/replace of a child row referencing a missing parent was committed instead of aborting — the lone hole in the "FK violations abort the whole write" contract that already held for NOT NULL / CHECK / UNIQUE.
The write now runs conn.pragma_update(None, "foreign_keys", true) right after opening the connection and before write_transaction's BEGIN IMMEDIATE (the pragma is a no-op inside a transaction, so it has to be set beforehand). An FK violation then lands on the existing ConstraintViolation path: abort errors with the offending row/column, skip does NOT swallow it (is_uniqueness_conflict is false for SQLITE_CONSTRAINT_FOREIGNKEY), and the transaction rolls back leaving the database byte-identical.
Added append_enforces_foreign_keys_and_rolls_back_dangling_child: a children→parents FK schema where appending parent_id = 999 aborts and leaves the file byte-identical, skip still refuses it, and a child referencing the existing parent (id 1) is written normally. Full gates green (808 rust lib tests, 445 frontend).
| // Memory gate: exact row count (cancellable) + sampled average row cost. | ||
| let n_rows: u64 = conn.query_row(&format!("SELECT COUNT(*) FROM {from}"), [], |r| { | ||
| r.get::<_, i64>(0) | ||
| })? as u64; |
There was a problem hiding this comment.
The editable import computes the memory gate from this COUNT(*) and sample, then later runs a separate SELECT * without holding a read transaction. If another connection adds many/large rows between those statements, the import can bypass the memory threshold and materialize far more data than the estimate allowed; it can also copy rows from a different snapshot than the schema/count shown. Wrap the count/sample/stream in the same read transaction (or otherwise pin a snapshot) before using the estimate to authorize the import.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 160aa46.
import_table now opens a single deferred read transaction (conn.unchecked_transaction()) and runs the schema read, the exact COUNT(*), the sampled-cost SELECT, and the streaming copy inside it, so the memory gate and the copy observe one pinned snapshot. The snapshot is taken at the first read and held until the copy finishes — a SHARED lock that blocks writers in rollback-journal mode, a stable read view in WAL mode. The authorizer already permits transaction control on the read-only handle (AuthAction::Transaction/Savepoint are allowed in safe_query::authorize), so no guard change was needed. The transaction is dropped right after the last row so the lock is released before the in-memory document is assembled.
Regression test import_reads_one_consistent_snapshot_across_a_concurrent_write reproduces the exact sequence (open_guarded → unchecked_transaction → COUNT → later read) and asserts a commit interleaved after the count is invisible to the pinned read (2 rows, not 4), then visible again once the transaction ends.
| patchDbExportForm: (patch) => { | ||
| const st = get().dbExport; | ||
| if (!st) return; | ||
| set({ dbExport: { ...st, form: { ...st.form, ...patch }, result: null, error: null } }); | ||
| if (dbPreviewTimer !== null) clearTimeout(dbPreviewTimer); | ||
| dbPreviewTimer = setTimeout(() => { |
There was a problem hiding this comment.
When the export form changes while a preview invoke is in flight, this keeps the old preview and does not advance dbPreviewToken until the debounced preview starts. An earlier response can therefore populate a preview for the previous path/mapping, leaving the Export button enabled with stale blockers/failure counts for the new form until the next preview finishes. Clear the preview or bump the token synchronously when patching the form so users cannot start an export without the preview matching the current spec.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 160aa46.
patchDbExportForm now invalidates the preview synchronously when the form changes: it bumps dbPreviewToken and clears preview (setting previewLoading: true) in the same set, before scheduling the debounced refresh. Bumping the token makes any preview invoke already in flight fail its token !== dbPreviewToken staleness guard when it resolves, so an earlier response can no longer repopulate a preview for the previous path/mapping. Clearing preview disables the Export button (runDbExport/canRunExport require a preview) until a fresh preview for the new form lands, so a user can never start an export whose blockers/failure counts were measured against a different spec.
Two vitest cases in useStore.test.ts cover it: one asserts an existing preview is dropped synchronously on a form change (Export disabled, no invoke yet), and one holds a preview invoke open, changes the form, then resolves the stale response and asserts it is ignored (preview stays null).
… preview Two Codex P2 fixes on PR #48 (feature f35 SQLite browser): - db_browser::import_table now runs its memory gate (exact count + sample) and the streaming copy inside one deferred read transaction, so both observe a single pinned snapshot. Previously a concurrent writer between the count and the stream could let the copy exceed the authorized memory estimate, or copy rows from a different snapshot than the schema/count the gate reported. - useStore.patchDbExportForm now invalidates the current preview synchronously: it bumps dbPreviewToken and clears preview (setting previewLoading) when the form changes. An in-flight preview invoke for the old form can no longer pass its staleness guard and repopulate a preview for the previous path/mapping, so the Export button stays disabled until a fresh preview for the new form lands. Tests: adds import_reads_one_consistent_snapshot_across_a_concurrent_write (Rust) and two useStore preview-invalidation cases (vitest). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The export write connection set busy_timeout but never enabled foreign-key enforcement. SQLite's per-connection default is foreign_keys = OFF, so an appended/replaced child row referencing a missing parent was COMMITTED instead of failing and rolling back — the one gap in the "FK violations abort the whole write" contract that already held for NOT NULL / CHECK / UNIQUE. Enable `PRAGMA foreign_keys = ON` in run_export right after opening the connection and before write_transaction's BEGIN IMMEDIATE (the pragma is a no-op inside a transaction). An FK violation now reaches the existing ConstraintViolation path: abort errors with the row/column, skip does not swallow it (not a uniqueness conflict), and the transaction rolls back leaving the database byte-identical. Adds append_enforces_foreign_keys_and_rolls_back_dangling_child covering the dangling-child abort (byte-identical), skip refusal, and a valid reference succeeding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # src/store/useStore.test.ts
1 participant
Summary
F35 adds a local database browser and database export to CEESVEE, built on a read-only, authorizer-guarded SafeQueryEngine core. SQLite this cycle.
Read side (browse & open) — open a SQLite file (
.db/.sqlite/.sqlite3/.db3) from the palette ("Open database…"), by opening the file, or via drag-and-drop. Browse tables and views (columns with declared type / NOT NULL / defaults / primary keys, plus indexes, foreign keys, WITHOUT ROWID flags, row-count estimates and bounded preview rows). Open any table or view as an indexed read-only document — paged straight out of the database through a newBacking::Virtualprovider that never copies a large table into memory — or import it into a fully editable document behind a memory check with an explicit "import anyway". External change (rows viaPRAGMA data_version, schema via a hash) is detected and offers a reload.Write side (export) — export the active document into a SQLite table (palette → "Export to database…"): create a new table, append to a compatible existing one, or replace one after explicit confirmation. A preview shows the resolved document-column → SQL-type mapping (from the F31 declared schema, TEXT by default, with per-column name/type/PK overrides) plus a bounded scan of cells that would fail to convert. The write runs in one transaction that rolls back completely on any failure, under an explicit primary-key conflict policy (abort / skip / replace), and never ALTERs an existing table's schema.
Safety invariants — every database read goes through a read-only connection with an SQLite authorizer restricting access to the files the user explicitly opened; result limits, a progress handler for cancellation, and revision-guarded snapshot semantics on the document virtual table. No arbitrary SQL/extension/exec surface; no data leaves the device.
Tests
cargo test --lib, the local--lib-only path with the comctl32 manifest embed; CI/MSVC runs the full suite). Includes the SafeQueryEngine snapshot/byte-budget/live-strategy tests, the virtual-rows windowed/coalesced read tests, and the db-export conversion/transaction/PK-policy tests.dbExport.test.ts.--check,tsc --noEmit, vitest, vite build;cargo fmt --all --check,clippy --all-targets --all-features -D warnings.Acceptance mapping
db_schema/DbSchemaInfo; DatabaseDialog renders tables/views, columns, indexes, FKs, row estimates, preview.start_db_open_table→Backing::Virtualwindowed provider (bounded block reads, coalesced scattered reads).start_db_import_tablewith the force gate; a spilled/indexed import opens unsaved.db_refresh_probe/db_doc_refresh_probe(PRAGMA data_version+ schema hash) →DbRefreshStatus.ApprovedSourcesregistry + SQLite authorizer.db_exportpreview (mapping + bounded failure scan).STACKED PR — review delta only
This PR is stacked on
feat/f34-excel(its base). Review only the delta versusfeat/f34-excel; the six commits here (da98416..e126f30) are the F35-only changes. The integration commit resolves rebase conflicts as pure additive unions (invoke_handler +.manage()in lib.rs;Backing::VirtualalongsideBacking::Columnarin document.rs; dto/types/tauri/commandDefs/useStore/App.tsx; CHANGELOG.md and README.md keep all chain bullets with F35 after existing ones), regeneratesCargo.lockfor the unionedCargo.toml(arrow/parquet/calamine from F32/F34 +rusqlitefor F35), and initializes the newdisplay_namefield in F32'sfrom_columnarconstructor.DuckDB — out of scope, by design
DuckDB support is deliberately out of scope this cycle. The only viable bundling path is a vendored
libduckdb, whose C++ build does not fit the 6 GB MinGW development machine (it OOMs the linker under GNU ld). SQLite ships now viarusqlite(bundled), covering the database-browser and export acceptance criteria; a DuckDB backend can slot behind the sameVirtualRows/ SafeQueryEngine seams later without reworking the UI or command surface.🤖 Generated with Claude Code