Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,26 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
files the user explicitly opened. _SQLite only this cycle — DuckDB support is
deliberately out of scope (its bundled C++ library cannot be built on the
low-memory MinGW development machine)._
- **Multi-facet exploration** (F39; toolbar → "Facets", palette → "Toggle
facets"): explore several dimensions at once, beyond the single-column
explorer. Add facet panels for any column — value counts (top-N with a search
box for high-cardinality columns), a numeric or date histogram with a range
selection, true/false, blank/null-token/invalid/value nullability, or a
semantic type — plus row-level status facets: diagnostics, validation
(cross-column rules and advisory schema issues), duplicate group membership,
and bookmarked/flagged/tagged. Several facets are active at once: the grid row
view is the AND across panels and the OR (with a per-facet include/exclude
toggle) among the selected values inside one, and every facet's counts are
recomputed against the population filtered by all the _other_ facets — classic
faceted search — so the counts always reflect the current cross-filter.
Reorder, pin, collapse, or remove panels; copy a facet's values and counts to
the clipboard; save the facet configuration inside a named view (restored with
the view); and convert the active facets to the standard filter builder in one
step (a one-way conversion; facets with no filter equivalent are reported).
Counts on a very large indexed document may be estimated from a leading sample
(marked as such); the applied row filter is always exact. Faceting is
non-destructive — it never dirties the document — and integrates with the
existing row-view pipeline, so visible-row export respects the facet filter.
- **Row bookmarks, tags & notes** (F40): mark and annotate records without
touching the source data. Star or flag a row, apply multiple named tags
(a per-document tag namespace with usage counts), and attach a row note or
Expand Down
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,21 @@ and faithful on large, real-world delimited files.**
blanks, distinct counts (exact, or estimated once cardinality explodes), top
values, numeric quartiles, date extremes, and text-length stats — over all
rows or just the visible ones, with click-to-filter straight from the panel.
- **Multi-facet exploration** — explore several dimensions at once, beyond the
single-column explorer: add facet panels for value counts (top-N with a
search box for high-cardinality columns), a numeric or date histogram with a
range selection, true/false, blank / null-token / invalid nullability, a
semantic type, or row-level status (diagnostics, validation, duplicate group,
and bookmarked / flagged / tagged). Several facets stay active at once — the
grid row view is the AND across panels and the OR (with a per-facet
include/exclude toggle) among the selected values inside one — and every
facet's counts recompute against the population filtered by all the _other_
facets (classic faceted search). Reorder, pin, collapse, or remove panels;
copy a facet's values and counts; save the facet configuration inside a named
view; and convert the active facets to the standard filter builder in one
step. Counts on a very large indexed document may be estimated from a leading
sample (marked as such) while the applied row filter stays exact; faceting
never dirties the document, and visible-row export respects the facet filter.
- **File profiles** — save delimiter / encoding / header choices, expected
columns, and validation rules (required, unique, type, regex, numeric range)
under a name matched to file patterns; matching files suggest — or with
Expand Down
159 changes: 159 additions & 0 deletions src-tauri/src/commands.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ use crate::excel::{
self, ExcelImportOptions, ExcelImportPreview, ExcelInspectCache, ExcelPreviewCache,
SheetSource, WorkbookInfo,
};
use crate::facets::{
FacetConfig, FacetConversion, FacetInputs, FacetKind, FacetResultSet, StatusInput,
};
use crate::follow::{self, FollowRegistry};
use crate::groupby::{self, GroupByPreview, GroupBySpec};
use crate::highlight::{
Expand Down Expand Up @@ -3091,6 +3094,162 @@ pub async fn start_highlight_report(
Ok(job_id)
}

// ----- multi-facet exploration (F39) -----------------------------------------

/// Snapshot the status-facet row memberships the config asks for, from the live
/// analysis caches (cloned so the engine never holds a cache lock while it
/// scans). Only the status facets actually present in `config` are resolved;
/// any that cannot be sourced are left `None`, so the engine renders them
/// `unresolved` (the UI explains what scan to run). Value facets read the
/// document directly and need no inputs.
///
/// * annotation — always resolvable: the F40 marks are re-resolved against the
/// current document (a matched record IS the current absolute row).
/// * diagnostics — resolvable once a diagnostics report (F02) is cached.
/// * validation — always resolvable: the document's advisory schema issues
/// (F31) are folded with any cached cross-column rules (F27).
/// * duplicate — resolvable when the caller passes the dedup spec/scope (F05);
/// the dedup cache stores only a report, so the rows are recomputed exactly.
#[allow(clippy::too_many_arguments)]
fn facet_inputs(
doc_id: u64,
doc: &Document,
config: &FacetConfig,
dedup: Option<&DedupSpec>,
dedup_scope: Option<&ExportScope>,
annotations: &State<'_, AnnotationRegistry>,
diagnostics_cache: &State<'_, DiagnosticsCache>,
crossval_cache: &State<'_, CrossValCache>,
) -> AppResult<FacetInputs> {
let has = |kind: FacetKind| config.facets.iter().any(|f| f.kind == kind);
let mut inputs = FacetInputs::default();

if has(FacetKind::Annotation) {
let source = DocumentSource::new(doc);
let idx = annotations.try_with(doc_id, |ann| ann.mark_index(&source, None))?;
inputs.annotation = Some(StatusInput::from_marks(&idx));
}
if has(FacetKind::Diagnostics) {
if let Some(report) = diagnostics_cache.get(doc_id) {
inputs.diagnostics = Some(StatusInput::from_diagnostics(doc, &report)?);
}
}
if has(FacetKind::Validation) {
let rules = crossval_cache
.get(doc_id)
.map(|(rules, _)| rules)
.unwrap_or_default();
inputs.validation = Some(StatusInput::from_validation(
doc,
&rules,
doc.schema_issues(),
)?);
}
if has(FacetKind::Duplicate) {
if let (Some(spec), Some(scope)) = (dedup, dedup_scope) {
inputs.duplicate = Some(StatusInput::from_duplicates(doc, spec, scope)?);
}
}
Ok(inputs)
}

/// Compute every facet's cross-filtered bucket counts against the current
/// document (F39). Read-only: never mutates or dirties the document, so it is
/// safe to recompute on every selection change. Guarded by `expected_revision`.
/// Counts on a very large indexed document may be estimated from a leading
/// sample (`FacetResultSet::sampled`); the applied filter is always exact.
#[allow(clippy::too_many_arguments)]
#[tauri::command]
pub fn compute_facets(
doc_id: u64,
config: FacetConfig,
expected_revision: u64,
dedup: Option<DedupSpec>,
dedup_scope: Option<ExportScope>,
state: Db<'_>,
annotations: State<'_, AnnotationRegistry>,
diagnostics_cache: State<'_, DiagnosticsCache>,
crossval_cache: State<'_, CrossValCache>,
) -> AppResult<FacetResultSet> {
read_doc(&state, doc_id, |doc| {
doc.check_revision(expected_revision)?;
let inputs = facet_inputs(
doc_id,
doc,
&config,
dedup.as_ref(),
dedup_scope.as_ref(),
&annotations,
&diagnostics_cache,
&crossval_cache,
)?;
crate::facets::compute(doc, &config, &inputs, None)
})
}

/// Drive the grid row view from the current facet selection (F39): an exact
/// full-document scan of the matching absolute rows, handed to the existing
/// row-filter pipeline so view sort, scoped previews and visible-row export all
/// compose. When no facet is active the facet-driven filter is cleared. A view
/// operation — never enters the undo stack and never dirties the document.
#[allow(clippy::too_many_arguments)]
#[tauri::command]
pub fn apply_facets(
doc_id: u64,
config: FacetConfig,
expected_revision: u64,
dedup: Option<DedupSpec>,
dedup_scope: Option<ExportScope>,
state: Db<'_>,
annotations: State<'_, AnnotationRegistry>,
diagnostics_cache: State<'_, DiagnosticsCache>,
crossval_cache: State<'_, CrossValCache>,
) -> AppResult<DocumentMeta> {
let handle = doc_handle(&state, doc_id)?;
let mut doc = handle.write().map_err(poisoned)?;
doc.check_revision(expected_revision)?;
if !config.any_active() {
doc.clear_filter()?;
return Ok(doc.meta());
}
let inputs = facet_inputs(
doc_id,
&doc,
&config,
dedup.as_ref(),
dedup_scope.as_ref(),
&annotations,
&diagnostics_cache,
&crossval_cache,
)?;
match crate::facets::narrowing_rows(&doc, &config, &inputs)? {
Some(rows) => doc.set_filter(rows)?,
// A selection is active, but every active facet is unresolved (e.g. a
// saved view referencing a deleted column, or a status facet with no
// cached scan). Applying `matching_rows` here would pass every row and
// install an all-rows filter that falsely marks the document filtered.
// Clear the facet-driven view instead — the facet result already reports
// the facet as unresolved, and it must not narrow anything.
None => doc.clear_filter()?,
}
Ok(doc.meta())
}

/// Convert the active facets to the standard filter-builder tree (F39): a
/// deliberately one-way, lossy conversion. Facets with no faithful column-filter
/// equivalent (semantic, the four status facets, some nullability/boolean
/// exclusions) are reported in `dropped` so the UI can explain what it left out.
#[tauri::command]
pub fn convert_facets_to_filter(
doc_id: u64,
config: FacetConfig,
state: Db<'_>,
) -> AppResult<FacetConversion> {
read_doc(&state, doc_id, |doc| {
Ok(crate::facets::to_filter_group(doc, &config))
})
}

#[tauri::command]
pub fn get_meta(doc_id: u64, state: Db<'_>) -> AppResult<DocumentMeta> {
read_doc(&state, doc_id, |doc| Ok(doc.meta()))
Expand Down
Loading
Loading