Skip to content

Avoid empty results table in main DB for split raw data storage#405

Open
astafan8 wants to merge 1 commit into
feature/split-raw-data-sqlitefrom
feature/split-raw-data-no-empty-table
Open

Avoid empty results table in main DB for split raw data storage#405
astafan8 wants to merge 1 commit into
feature/split-raw-data-sqlitefrom
feature/split-raw-data-no-empty-table

Conversation

@astafan8

Copy link
Copy Markdown
Owner

Summary

This is a stacked change on top of microsoft#8219 (targets feature/split-raw-data-sqlite, not main) that removes the empty results table from the main database for split raw data storage.

Previously, even with split raw data storage enabled, an empty results table was still created in the main database purely to satisfy code paths that assumed the table exists. That was flagged in microsoft#8219 as unclean/hacky, and becomes more confusing once raw data can be written to non-SQLite backends. This PR removes that table entirely: the main DB then holds only metadata.

What changes

No results table in the main DB

  • When raw data storage is enabled, the run is created with create_run_table=False and parameters are added with insert_into_results_table=False. This mirrors exactly how DataSetInMem (the netcdf-backed dataset) already records runs without a results table.

Identifying split-storage runs (disambiguation)

  • The empty table was previously used to tell a DataSet apart from a DataSetInMem (both otherwise "have no results table") in _get_datasetprotocol_from_guid.
  • Replaced with a positive marker: a dedicated raw_data_db_path column in the runs table, recorded at dataset creation (so even an unstarted dataset is correctly identified). This is future-proof for other raw-data backends.

raw_data_db_path is now an internal storage detail

  • It is stored in the runs table but excluded from the user-facing metadata dict (via get_metadata_from_run_id), read/written through get_raw_data_db_path_for_run / set_raw_data_db_path_for_run.
  • This also fixes it leaking into metadata comparisons (the_same_dataset_as) and being copied into extracted/exported databases.

Features that depended on the empty table — now handled explicitly

  • Counting (number_of_results / __len__): return 0 when the results table does not exist (e.g. a pristine, not-yet-started split dataset).
  • Subscribers: a subscription requested before the dataset is started (before the raw table exists) is deferred and materialised at start time. This is semantically correct (there is nothing to observe before start) and fixes trigger creation.
  • Loading: load_by_id/load_by_guid/load_by_counter correctly reconstruct split datasets (started or not) from the raw_data_db_path marker.
  • Management helpers (purge_orphaned_datasets / cleanup_datasets): restricted to started runs only, since an unstarted run records an intended path but never creates the file.

Verification

Validated by temporarily enabling split raw data storage globally for the whole tests/dataset suite (autouse fixture) and iterating until only expected/inherent failures remained:

  • Full tests/dataset suite in default mode: 1064 passed, 35 skipped — no regressions.
  • Full suite with split globally enabled: 9 failures remain, all expected/inherent (down from 23 while developing this change):
    • 3 tests explicitly assert disabled-mode behaviour (artifact of the global-enable harness).
    • 2 low-level tests call queries.get_parameter_data(ds.conn, ...) directly, bypassing the DataSet API (already documented expected failures in feat: split raw data storage into per-dataset SQLite files microsoft/Qcodes#8219).
    • 3 test_db_overview tests + 1 test_guids_from_dir assert main-DB row counts / scan for DB files — these are inherent to the feature (data isn't in the main DB) and fail with the empty-table approach too; they are not regressions from removing the table.
  • test_raw_data_storage.py: all pass, with new tests for subscribe-before-start deferral, counting before start, absence of a main-DB results table, and extract not carrying over raw_data_db_path.
  • Pyright: 0 errors. Ruff check + format: pass.

Note

The autouse "global enable" fixture used for validation is not included in this PR — it was only a local harness.

Previously a results table was still created (empty) in the main database
for split-storage datasets, purely to satisfy code paths that assumed the
table exists. This removes that table entirely so the main DB holds only
metadata - cleaner and future-proof for non-sqlite raw data backends.

Changes:
- Do not create the results table in the main DB when raw storage is
  enabled (create_run_table=False, insert_into_results_table=False),
  mirroring how DataSetInMem records runs.
- Identify split-storage runs via a dedicated 'raw_data_db_path' column in
  the runs table, recorded at dataset creation. This disambiguates them
  from DataSetInMem runs (which also have no results table) in
  _get_datasetprotocol_from_guid, replacing the old reliance on the empty
  table's presence.
- Keep 'raw_data_db_path' out of the user-facing metadata dict (read/write
  via get/set_raw_data_db_path_for_run); this also stops it leaking into
  metadata comparisons and extract/export copies.
- number_of_results/__len__ return 0 when the results table does not exist
  (e.g. a pristine, not-yet-started split dataset).
- Defer subscriptions requested before the dataset is started until start
  time, when the raw data table exists (fixes subscriber trigger creation).
- Restrict management queries (purge/cleanup) to started runs only.
- Docs + tests updated; add tests for subscribe-before-start, count before
  start, no-main-table, and extract not carrying over raw_data_db_path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant