diff --git a/CHANGELOG.md b/CHANGELOG.md index ae3cbac..0e216b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,6 +87,26 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). files the user explicitly opened. _SQLite only this cycle — DuckDB support is deliberately out of scope (its bundled C++ library cannot be built on the low-memory MinGW development machine)._ +- **Multi-facet exploration** (F39; toolbar → "Facets", palette → "Toggle + facets"): explore several dimensions at once, beyond the single-column + explorer. Add facet panels for any column — value counts (top-N with a search + box for high-cardinality columns), a numeric or date histogram with a range + selection, true/false, blank/null-token/invalid/value nullability, or a + semantic type — plus row-level status facets: diagnostics, validation + (cross-column rules and advisory schema issues), duplicate group membership, + and bookmarked/flagged/tagged. Several facets are active at once: the grid row + view is the AND across panels and the OR (with a per-facet include/exclude + toggle) among the selected values inside one, and every facet's counts are + recomputed against the population filtered by all the _other_ facets — classic + faceted search — so the counts always reflect the current cross-filter. + Reorder, pin, collapse, or remove panels; copy a facet's values and counts to + the clipboard; save the facet configuration inside a named view (restored with + the view); and convert the active facets to the standard filter builder in one + step (a one-way conversion; facets with no filter equivalent are reported). + Counts on a very large indexed document may be estimated from a leading sample + (marked as such); the applied row filter is always exact. Faceting is + non-destructive — it never dirties the document — and integrates with the + existing row-view pipeline, so visible-row export respects the facet filter. - **Row bookmarks, tags & notes** (F40): mark and annotate records without touching the source data. Star or flag a row, apply multiple named tags (a per-document tag namespace with usage counts), and attach a row note or diff --git a/README.md b/README.md index 86d41a3..d7f4f06 100644 --- a/README.md +++ b/README.md @@ -217,6 +217,21 @@ and faithful on large, real-world delimited files.** blanks, distinct counts (exact, or estimated once cardinality explodes), top values, numeric quartiles, date extremes, and text-length stats — over all rows or just the visible ones, with click-to-filter straight from the panel. +- **Multi-facet exploration** — explore several dimensions at once, beyond the + single-column explorer: add facet panels for value counts (top-N with a + search box for high-cardinality columns), a numeric or date histogram with a + range selection, true/false, blank / null-token / invalid nullability, a + semantic type, or row-level status (diagnostics, validation, duplicate group, + and bookmarked / flagged / tagged). Several facets stay active at once — the + grid row view is the AND across panels and the OR (with a per-facet + include/exclude toggle) among the selected values inside one — and every + facet's counts recompute against the population filtered by all the _other_ + facets (classic faceted search). Reorder, pin, collapse, or remove panels; + copy a facet's values and counts; save the facet configuration inside a named + view; and convert the active facets to the standard filter builder in one + step. Counts on a very large indexed document may be estimated from a leading + sample (marked as such) while the applied row filter stays exact; faceting + never dirties the document, and visible-row export respects the facet filter. - **File profiles** — save delimiter / encoding / header choices, expected columns, and validation rules (required, unique, type, regex, numeric range) under a name matched to file patterns; matching files suggest — or with diff --git a/src-tauri/src/commands.rs b/src-tauri/src/commands.rs index e78d9af..034dd84 100644 --- a/src-tauri/src/commands.rs +++ b/src-tauri/src/commands.rs @@ -45,6 +45,9 @@ use crate::excel::{ self, ExcelImportOptions, ExcelImportPreview, ExcelInspectCache, ExcelPreviewCache, SheetSource, WorkbookInfo, }; +use crate::facets::{ + FacetConfig, FacetConversion, FacetInputs, FacetKind, FacetResultSet, StatusInput, +}; use crate::follow::{self, FollowRegistry}; use crate::groupby::{self, GroupByPreview, GroupBySpec}; use crate::highlight::{ @@ -3091,6 +3094,162 @@ pub async fn start_highlight_report( Ok(job_id) } +// ----- multi-facet exploration (F39) ----------------------------------------- + +/// Snapshot the status-facet row memberships the config asks for, from the live +/// analysis caches (cloned so the engine never holds a cache lock while it +/// scans). Only the status facets actually present in `config` are resolved; +/// any that cannot be sourced are left `None`, so the engine renders them +/// `unresolved` (the UI explains what scan to run). Value facets read the +/// document directly and need no inputs. +/// +/// * annotation — always resolvable: the F40 marks are re-resolved against the +/// current document (a matched record IS the current absolute row). +/// * diagnostics — resolvable once a diagnostics report (F02) is cached. +/// * validation — always resolvable: the document's advisory schema issues +/// (F31) are folded with any cached cross-column rules (F27). +/// * duplicate — resolvable when the caller passes the dedup spec/scope (F05); +/// the dedup cache stores only a report, so the rows are recomputed exactly. +#[allow(clippy::too_many_arguments)] +fn facet_inputs( + doc_id: u64, + doc: &Document, + config: &FacetConfig, + dedup: Option<&DedupSpec>, + dedup_scope: Option<&ExportScope>, + annotations: &State<'_, AnnotationRegistry>, + diagnostics_cache: &State<'_, DiagnosticsCache>, + crossval_cache: &State<'_, CrossValCache>, +) -> AppResult { + let has = |kind: FacetKind| config.facets.iter().any(|f| f.kind == kind); + let mut inputs = FacetInputs::default(); + + if has(FacetKind::Annotation) { + let source = DocumentSource::new(doc); + let idx = annotations.try_with(doc_id, |ann| ann.mark_index(&source, None))?; + inputs.annotation = Some(StatusInput::from_marks(&idx)); + } + if has(FacetKind::Diagnostics) { + if let Some(report) = diagnostics_cache.get(doc_id) { + inputs.diagnostics = Some(StatusInput::from_diagnostics(doc, &report)?); + } + } + if has(FacetKind::Validation) { + let rules = crossval_cache + .get(doc_id) + .map(|(rules, _)| rules) + .unwrap_or_default(); + inputs.validation = Some(StatusInput::from_validation( + doc, + &rules, + doc.schema_issues(), + )?); + } + if has(FacetKind::Duplicate) { + if let (Some(spec), Some(scope)) = (dedup, dedup_scope) { + inputs.duplicate = Some(StatusInput::from_duplicates(doc, spec, scope)?); + } + } + Ok(inputs) +} + +/// Compute every facet's cross-filtered bucket counts against the current +/// document (F39). Read-only: never mutates or dirties the document, so it is +/// safe to recompute on every selection change. Guarded by `expected_revision`. +/// Counts on a very large indexed document may be estimated from a leading +/// sample (`FacetResultSet::sampled`); the applied filter is always exact. +#[allow(clippy::too_many_arguments)] +#[tauri::command] +pub fn compute_facets( + doc_id: u64, + config: FacetConfig, + expected_revision: u64, + dedup: Option, + dedup_scope: Option, + state: Db<'_>, + annotations: State<'_, AnnotationRegistry>, + diagnostics_cache: State<'_, DiagnosticsCache>, + crossval_cache: State<'_, CrossValCache>, +) -> AppResult { + read_doc(&state, doc_id, |doc| { + doc.check_revision(expected_revision)?; + let inputs = facet_inputs( + doc_id, + doc, + &config, + dedup.as_ref(), + dedup_scope.as_ref(), + &annotations, + &diagnostics_cache, + &crossval_cache, + )?; + crate::facets::compute(doc, &config, &inputs, None) + }) +} + +/// Drive the grid row view from the current facet selection (F39): an exact +/// full-document scan of the matching absolute rows, handed to the existing +/// row-filter pipeline so view sort, scoped previews and visible-row export all +/// compose. When no facet is active the facet-driven filter is cleared. A view +/// operation — never enters the undo stack and never dirties the document. +#[allow(clippy::too_many_arguments)] +#[tauri::command] +pub fn apply_facets( + doc_id: u64, + config: FacetConfig, + expected_revision: u64, + dedup: Option, + dedup_scope: Option, + state: Db<'_>, + annotations: State<'_, AnnotationRegistry>, + diagnostics_cache: State<'_, DiagnosticsCache>, + crossval_cache: State<'_, CrossValCache>, +) -> AppResult { + let handle = doc_handle(&state, doc_id)?; + let mut doc = handle.write().map_err(poisoned)?; + doc.check_revision(expected_revision)?; + if !config.any_active() { + doc.clear_filter()?; + return Ok(doc.meta()); + } + let inputs = facet_inputs( + doc_id, + &doc, + &config, + dedup.as_ref(), + dedup_scope.as_ref(), + &annotations, + &diagnostics_cache, + &crossval_cache, + )?; + match crate::facets::narrowing_rows(&doc, &config, &inputs)? { + Some(rows) => doc.set_filter(rows)?, + // A selection is active, but every active facet is unresolved (e.g. a + // saved view referencing a deleted column, or a status facet with no + // cached scan). Applying `matching_rows` here would pass every row and + // install an all-rows filter that falsely marks the document filtered. + // Clear the facet-driven view instead — the facet result already reports + // the facet as unresolved, and it must not narrow anything. + None => doc.clear_filter()?, + } + Ok(doc.meta()) +} + +/// Convert the active facets to the standard filter-builder tree (F39): a +/// deliberately one-way, lossy conversion. Facets with no faithful column-filter +/// equivalent (semantic, the four status facets, some nullability/boolean +/// exclusions) are reported in `dropped` so the UI can explain what it left out. +#[tauri::command] +pub fn convert_facets_to_filter( + doc_id: u64, + config: FacetConfig, + state: Db<'_>, +) -> AppResult { + read_doc(&state, doc_id, |doc| { + Ok(crate::facets::to_filter_group(doc, &config)) + }) +} + #[tauri::command] pub fn get_meta(doc_id: u64, state: Db<'_>) -> AppResult { read_doc(&state, doc_id, |doc| Ok(doc.meta())) diff --git a/src-tauri/src/facets.rs b/src-tauri/src/facets.rs new file mode 100644 index 0000000..63825dc --- /dev/null +++ b/src-tauri/src/facets.rs @@ -0,0 +1,2407 @@ +//! Multi-facet exploration engine (F39). +//! +//! A *facet* is one dimension of a document the user can slice by. Several +//! facets are active at once; the population is the AND across facet panels and +//! the OR among the selected values inside one panel, with a per-value +//! include/exclude mode. Each facet's bucket counts are recomputed against the +//! population filtered by *all the other* facets — classic faceted search — so +//! selecting a city updates the age histogram but not the city counts. +//! +//! # Design +//! +//! The whole engine is a **pure function of a [`Document`] plus a +//! [`FacetInputs`] bundle**, so it is entirely unit-testable and never touches +//! Tauri state. Value facets (text / number / date / boolean / nullability / +//! semantic) read cell values (and, when a column has a declared F31 schema, +//! classify through it). The four *status* facets (diagnostics, validation, +//! duplicate, annotation) are row-level: their per-row membership is resolved by +//! the command layer from the live analysis caches (or recomputed) and handed in +//! as [`FacetInputs`], mirroring how F42 highlighting snapshots caches into an +//! `AnalysisContext`. Convenience constructors ([`StatusInput::from_marks`], +//! [`StatusInput::from_diagnostics`], …) build those from the standard report +//! types so the command layer stays trivial. +//! +//! # Cross-filter counting +//! +//! [`compute`] makes a single streaming pass. For each row it evaluates every +//! facet's selection predicate and counts the failures. A row joins the final +//! population when **no** facet rejects it; a facet's own counts include a row +//! when **every other** facet accepts it (i.e. either nothing fails, or the only +//! failing facet is this one). That is the standard faceting optimisation and +//! yields exact "counts reflect all other facets' selections" semantics in one +//! pass. +//! +//! # Non-destructive integration +//! +//! Applying facets is [`matching_rows`]: an exact full-document scan returning +//! the absolute row indices the current selection admits, ready to hand to +//! [`Document::set_filter`]. It produces the same kind of row view a plain +//! filter does, so the F12 view sort, scoped export and visible-row export all +//! compose for free. Computing counts and applying facets never mutate the +//! document and never dirty it. [`to_filter_group`] converts the (convertible) +//! facets to the standard filter-builder representation — a deliberately +//! one-way, lossy conversion (semantic / status facets have no column-filter +//! equivalent and are reported as dropped). +//! +//! Counts may be estimated from a leading sample on very large indexed +//! documents ([`FacetResultSet::sampled`] flags it); the *applied* filter is +//! always exact over the whole document. + +use std::collections::{HashMap, HashSet}; + +use serde::{Deserialize, Serialize}; + +use crate::analyze; +use crate::annotations::MarkIndex; +use crate::crossval::{self, CrossRule}; +use crate::dedup::{self, DedupSpec}; +use crate::diagnostics::{self, DiagnosticsReport}; +use crate::document::Document; +use crate::dto::{Conjunction, ExportScope, FilterCondition, FilterGroup, FilterNode, FilterOp}; +use crate::error::{AppError, AppResult}; +use crate::job::JobCtx; +use crate::schema::{self, CellState, ColumnSchema, NumericCell, SchemaIssue, TypedValue}; +use crate::semantic::{self, SemanticType}; + +/// Default number of top text values returned when a facet does not override it. +pub const DEFAULT_TEXT_TOP_N: usize = 20; +/// Hard cap on the distinct text values one facet tracks in memory. Beyond this +/// the count map stops admitting *new* values (existing ones keep counting) and +/// the result is flagged [`FacetResult::truncated`] — the "never a full value +/// dump" memory bound for high-cardinality columns. +pub const MAX_TEXT_VALUES: usize = 10_000; +/// Default histogram bin count for numeric / date facets. +pub const DEFAULT_BINS: usize = 20; +/// Upper bound on histogram bins. +pub const MAX_BINS: usize = 100; +/// Leading rows scanned for COUNTS on an indexed (read-only) document; beyond +/// this the counts are estimated and flagged. The applied filter is unaffected. +pub const FACET_SAMPLE_ROWS: usize = 200_000; +/// Progress/cancel granularity for the streaming passes. +const CHUNK: u64 = 4096; +/// Status facets pack their positive categories into a `u64` bitmask, so at most +/// this many are tracked (extras are dropped — far more than any real report). +const MAX_STATUS_CATEGORIES: usize = 63; + +// =========================================================================== +// Public DTOs +// =========================================================================== + +/// The ten facet dimensions. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum FacetKind { + /// Distinct value counts (top-N + search for high-cardinality columns). + Text, + /// Numeric histogram + range selection. + Number, + /// Date range + coarse histogram. + Date, + /// True / false (+ blank / other) buckets. + Boolean, + /// Blank / null-token / invalid / value, via F31 [`schema::classify`]. + Nullability, + /// Matches a given [`SemanticType`] or not. + Semantic, + /// Row-level diagnostics status (F02). + Diagnostics, + /// Cross-column (F27) + advisory schema (F31) validation status. + Validation, + /// Duplicate-group membership (F05). + Duplicate, + /// Bookmarked / flagged / tagged (F40). + Annotation, +} + +impl FacetKind { + fn is_column_scoped(self) -> bool { + matches!( + self, + FacetKind::Text + | FacetKind::Number + | FacetKind::Date + | FacetKind::Boolean + | FacetKind::Nullability + | FacetKind::Semantic + ) + } +} + +/// Whether selected values keep (include) or remove (exclude) matching rows. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum FacetMode { + #[default] + Include, + Exclude, +} + +/// A continuous inclusive range selection for number / date facets. Bounds are +/// carried as strings so they parse under the column's declared schema (locale, +/// date formats) exactly like a filter-builder range condition. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct FacetRange { + #[serde(default)] + pub min: Option, + #[serde(default)] + pub max: Option, +} + +impl FacetRange { + fn is_empty(&self) -> bool { + blankless(&self.min).is_none() && blankless(&self.max).is_none() + } +} + +/// One facet's active selection: OR among `values`, plus an optional continuous +/// `range` (number / date), under an include/exclude `mode`. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct FacetSelection { + #[serde(default)] + pub mode: FacetMode, + /// Selected categorical value keys (text values; boolean / nullability / + /// semantic / status bucket keys). Unused for number / date facets. + #[serde(default)] + pub values: Vec, + /// Continuous range (number / date facets). + #[serde(default)] + pub range: FacetRange, +} + +impl FacetSelection { + fn value_set(&self) -> HashSet<&str> { + self.values.iter().map(String::as_str).collect() + } +} + +/// One facet panel's full specification: what it slices, its selection, its +/// display tuning and its (persisted, non-computational) panel layout. The +/// order of `FacetConfig::facets` is the panel order; `pinned` / `collapsed` / +/// `width` round-trip inside a saved F12 view. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct FacetSpec { + /// Stable per-panel id (the frontend's key; echoed into results). + pub id: String, + pub kind: FacetKind, + /// Stable logical column id (F12) for column-scoped facets; ignored by the + /// four status facets. + #[serde(default)] + pub column_id: Option, + /// The semantic type a [`FacetKind::Semantic`] facet tests against. + #[serde(default)] + pub semantic: Option, + #[serde(default)] + pub selection: FacetSelection, + /// Text facet: how many top values to return (default [`DEFAULT_TEXT_TOP_N`]). + #[serde(default)] + pub top_n: Option, + /// Text facet: case-insensitive substring narrowing the returned values. + #[serde(default)] + pub search: Option, + /// Number / date facet: histogram bin count (default [`DEFAULT_BINS`]). + #[serde(default)] + pub bins: Option, + // ----- panel layout (persisted, never affects counts) ----- + #[serde(default)] + pub pinned: bool, + #[serde(default)] + pub collapsed: bool, + #[serde(default)] + pub width: Option, +} + +/// The saved multi-facet configuration (extends a named F12 view). Ordered by +/// panel position. +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct FacetConfig { + #[serde(default)] + pub facets: Vec, +} + +impl FacetConfig { + /// Whether any facet carries an active selection (used to decide whether a + /// filter is even applied). + pub fn any_active(&self) -> bool { + self.facets.iter().any(|f| f.selection_is_active()) + } +} + +impl FacetSpec { + fn selection_is_active(&self) -> bool { + !self.selection.values.is_empty() || !self.selection.range.is_empty() + } +} + +/// One bucket of a facet result: a selectable value/category with its +/// cross-filtered count. +#[derive(Debug, Clone, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct FacetBucket { + /// Stable key the frontend echoes back in [`FacetSelection::values`]. + pub key: String, + /// Human label (equals `key` for text values). + pub label: String, + pub count: u64, + /// Whether this bucket is in the facet's current selection. + pub selected: bool, + /// Numeric/date histogram bins carry their edges so a bin click maps to a + /// range (timestamps for date facets). + #[serde(skip_serializing_if = "Option::is_none")] + pub lo: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub hi: Option, +} + +/// Observed extent and current selection of a number / date facet. +#[derive(Debug, Clone, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct RangeInfo { + /// Observed minimum over the scanned population (display string). + pub min: Option, + /// Observed maximum over the scanned population (display string). + pub max: Option, + /// The current selection bounds, echoed back verbatim. + pub selected_min: Option, + pub selected_max: Option, +} + +/// One facet's computed result: its bounded buckets plus metadata. +#[derive(Debug, Clone, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct FacetResult { + pub id: String, + pub kind: FacetKind, + #[serde(skip_serializing_if = "Option::is_none")] + pub column_id: Option, + pub mode: FacetMode, + /// Whether this facet is currently narrowing the population. + pub active: bool, + /// The facet's column/inputs could not be resolved (missing column after a + /// structural edit, or no cached data for a status facet) — it neither + /// filters nor produces counts. + pub unresolved: bool, + /// Counts are estimated from a sample (large indexed document). + pub sampled: bool, + /// Text facet only: the value map hit [`MAX_TEXT_VALUES`]; some low-count + /// values are not represented. + pub truncated: bool, + /// Text facet only: distinct values observed (may exceed returned buckets). + #[serde(skip_serializing_if = "Option::is_none")] + pub distinct: Option, + /// Bounded buckets (top-N + selected + search hits for text; the fixed set + /// otherwise). + pub buckets: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + pub range: Option, +} + +/// The full result of a facet computation over one document. +#[derive(Debug, Clone, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct FacetResultSet { + /// Document revision the counts were computed against. + pub revision: u64, + /// Rows in the composed (all-facets) population within the scanned range. + pub matched_rows: usize, + /// Total data rows in the document. + pub total_rows: usize, + /// Rows actually scanned (equals `total_rows` unless sampled). + pub scanned_rows: usize, + /// Any facet's counts are estimated from a sample. + pub sampled: bool, + pub facets: Vec, +} + +/// A facet that could not be represented as a column filter during +/// [`to_filter_group`] conversion. +#[derive(Debug, Clone, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct DroppedFacet { + pub id: String, + pub reason: String, +} + +/// Result of the one-way facets → filter-builder conversion. +#[derive(Debug, Clone, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct FacetConversion { + /// The equivalent filter tree (AND across facets). Empty group = match all. + pub filter: FilterGroup, + /// Facets that had no faithful column-filter equivalent and were omitted. + pub dropped: Vec, +} + +// =========================================================================== +// Status-facet inputs (resolved by the command layer from the analysis caches) +// =========================================================================== + +/// One positive category of a status facet: its key, label and the absolute +/// rows it covers (overlaps across categories are allowed — a row can carry +/// several tags or violate several rules). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct StatusCategory { + pub key: String, + pub label: String, + pub rows: Vec, +} + +/// Row-level membership for one status facet dimension. Rows in no positive +/// category fall into the synthesized `none_label` bucket (e.g. "clean", +/// "unique", "none") when one is set. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct StatusInput { + pub categories: Vec, + pub none_label: Option, +} + +impl StatusInput { + /// The annotation facet, wired from a resolved [`MarkIndex`] (F40). A matched + /// record number *is* the current absolute row, exactly as the annotation + /// row-filter and F42 highlighting treat it. + pub fn from_marks(index: &MarkIndex) -> StatusInput { + let to_rows = |records: &[u64]| records.iter().map(|&r| r as usize).collect::>(); + let mut categories = vec![ + StatusCategory { + key: "starred".into(), + label: "Bookmarked".into(), + rows: to_rows(&index.starred), + }, + StatusCategory { + key: "flagged".into(), + label: "Flagged".into(), + rows: to_rows(&index.flagged), + }, + ]; + for (tag, records) in &index.tagged { + categories.push(StatusCategory { + key: format!("tag:{tag}"), + label: format!("Tag: {tag}"), + rows: to_rows(records), + }); + } + StatusInput { + categories, + none_label: Some("No annotation".into()), + } + } + + /// The diagnostics facet, wired from a cached [`DiagnosticsReport`] (F02). + /// One category per row-filterable issue, rows recomputed against the current + /// document via [`diagnostics::issue_rows`]. + pub fn from_diagnostics(doc: &Document, report: &DiagnosticsReport) -> AppResult { + let mut categories = Vec::new(); + let mut seen: HashSet<&str> = HashSet::new(); + for issue in report.source.iter().chain(report.current.iter()) { + if !issue.row_filterable || !seen.insert(issue.id.as_str()) { + continue; + } + // A stale cache entry can name a column removed by a later structural + // edit (nothing invalidates the diagnostics cache on edit). Mirror the + // F42 highlight engine: skip such an issue so one dead category never + // fails the whole facet computation, degrading it to unresolved. + if let Ok(rows) = diagnostics::issue_rows(doc, &issue.id) { + categories.push(StatusCategory { + key: issue.id.clone(), + label: issue.title.clone(), + rows, + }); + } + } + Ok(StatusInput { + categories, + none_label: Some("No issues".into()), + }) + } + + /// The validation facet, wired from the cross-column rules (F27) and the + /// document's advisory schema issues (F31). Two categories: any cross-rule + /// violation, and any recorded schema-type issue. + pub fn from_validation( + doc: &Document, + rules: &[CrossRule], + schema_issues: &[SchemaIssue], + ) -> AppResult { + let mut categories = Vec::new(); + if !rules.is_empty() { + // Cached rules reference columns by name; a rename/delete makes + // `violating_rows` error on resolution. Mirror F42 highlighting + // (`.unwrap_or_default()`): a stale rule degrades the category to + // empty rather than failing the whole computation. + let rows = crossval::violating_rows(doc, rules, None).unwrap_or_default(); + categories.push(StatusCategory { + key: "crossval".into(), + label: "Cross-column violation".into(), + rows, + }); + } + let n = doc.n_rows(); + let mut schema_rows: Vec = schema_issues + .iter() + .map(|s| s.row) + .filter(|&r| r < n) + .collect(); + schema_rows.sort_unstable(); + schema_rows.dedup(); + categories.push(StatusCategory { + key: "schema".into(), + label: "Schema type issue".into(), + rows: schema_rows, + }); + Ok(StatusInput { + categories, + none_label: Some("Valid".into()), + }) + } + + /// The duplicate facet, wired from a dedup spec (F05): the rows in any + /// duplicate group, against the given scope. + pub fn from_duplicates( + doc: &Document, + spec: &DedupSpec, + scope: &ExportScope, + ) -> AppResult { + let rows = dedup::duplicate_row_indices(doc, spec, scope)?; + Ok(StatusInput { + categories: vec![StatusCategory { + key: "duplicate".into(), + label: "In a duplicate group".into(), + rows, + }], + none_label: Some("Unique".into()), + }) + } +} + +/// The status-facet row memberships passed into the engine. Each field is +/// populated only when a facet of that kind is present in the config; a missing +/// field renders its facet [`FacetResult::unresolved`]. `sampled` marks that the +/// underlying reports covered only a sample. +#[derive(Debug, Clone, Default)] +pub struct FacetInputs { + pub diagnostics: Option, + pub validation: Option, + pub duplicate: Option, + pub annotation: Option, + pub sampled: bool, +} + +impl FacetInputs { + fn for_kind(&self, kind: FacetKind) -> Option<&StatusInput> { + match kind { + FacetKind::Diagnostics => self.diagnostics.as_ref(), + FacetKind::Validation => self.validation.as_ref(), + FacetKind::Duplicate => self.duplicate.as_ref(), + FacetKind::Annotation => self.annotation.as_ref(), + _ => None, + } + } +} + +// =========================================================================== +// Public entry points +// =========================================================================== + +/// Compute every facet's cross-filtered bucket counts plus the composed +/// population size, in one streaming pass. Read-only; never mutates or dirties +/// the document. `ctx` (when present) reports progress and observes +/// cancellation. Counts on a large indexed document are estimated from a leading +/// sample and flagged. +pub fn compute( + doc: &Document, + config: &FacetConfig, + inputs: &FacetInputs, + ctx: Option<&JobCtx>, +) -> AppResult { + let scan_end = if doc.is_editable() { + doc.n_rows() + } else { + FACET_SAMPLE_ROWS.min(doc.n_rows()) + }; + compute_scan(doc, config, inputs, ctx, scan_end) +} + +/// Core of [`compute`] with an explicit scan bound. Rows `0..scan_end` are +/// counted; when `scan_end < n_rows` the counts are a leading-sample estimate +/// and every facet's `sampled` flag (value AND status) plus the top-level +/// [`FacetResultSet::sampled`] is set — status facets tally over the same window +/// and so are under-counted identically. Split out so the sampled path is +/// unit-testable without a multi-hundred-thousand-row indexed document. +fn compute_scan( + doc: &Document, + config: &FacetConfig, + inputs: &FacetInputs, + ctx: Option<&JobCtx>, + scan_end: usize, +) -> AppResult { + let n = doc.n_rows(); + let scan_end = scan_end.min(n); + let scan_sampled = scan_end < n; + + let mut built = build_facets(doc, config, inputs)?; + let has_range = built.iter().any(|b| matches!(b.eval, Eval::Range(_))); + + if let Some(c) = ctx { + let total = if has_range { scan_end * 2 } else { scan_end }; + c.set_total(total as u64); + } + + // Pre-pass: observe numeric/date extents so histogram edges are stable. + if has_range { + let mut pending = 0u64; + doc.visit_rows(0..scan_end, &mut |_, row| { + for b in built.iter_mut() { + if let Eval::Range(r) = &mut b.eval { + r.observe(row); + } + } + pending += 1; + if pending >= CHUNK { + if let Some(c) = ctx { + c.advance(pending)?; + } + pending = 0; + } + Ok(true) + })?; + if let Some(c) = ctx { + c.advance(pending)?; + } + for b in built.iter_mut() { + if let Eval::Range(r) = &mut b.eval { + r.finalize_bins(); + } + } + } + + // Counting pass with the classic all-other-facets-pass accounting. + let mut matched = 0usize; + let mut passes: Vec = Vec::with_capacity(built.len()); + let mut pending = 0u64; + doc.visit_rows(0..scan_end, &mut |i, row| { + passes.clear(); + let mut fail = 0usize; + let mut only = 0usize; + for (idx, b) in built.iter().enumerate() { + let p = b.eval.passes(i, row); + if !p { + fail += 1; + only = idx; + } + passes.push(p); + } + if fail == 0 { + matched += 1; + for b in built.iter_mut() { + b.eval.tally(i, row); + } + } else if fail == 1 { + built[only].eval.tally(i, row); + } + pending += 1; + if pending >= CHUNK { + if let Some(c) = ctx { + c.advance(pending)?; + } + pending = 0; + } + Ok(true) + })?; + if let Some(c) = ctx { + c.advance(pending)?; + } + + let facets: Vec = built + .into_iter() + .map(|b| b.finish(scan_sampled, inputs.sampled)) + .collect(); + + Ok(FacetResultSet { + revision: doc.revision(), + matched_rows: matched, + total_rows: n, + scanned_rows: scan_end, + sampled: scan_sampled || inputs.sampled, + facets, + }) +} + +/// Resolve the facet selection to the absolute row indices it admits, in source +/// order — the row view to hand to [`Document::set_filter`]. An **exact** +/// full-document scan (never sampled), so visible-row export respects the facet +/// filter. Inactive and unresolved facets never narrow the result. +pub fn matching_rows( + doc: &Document, + config: &FacetConfig, + inputs: &FacetInputs, +) -> AppResult> { + let built = build_facets(doc, config, inputs)?; + scan_matching(doc, &built) +} + +/// Like [`matching_rows`], but returns `None` when no facet actually narrows the +/// population — i.e. every *active* selection is unresolved (a deleted column, or +/// an unavailable status input). `FacetConfig::any_active` can be true in that +/// case, yet the only honest row view is "no facet filter": returning `Some(all +/// rows)` would falsely mark the document filtered. Callers use `None` to leave +/// the existing view / clear the facet filter instead of installing an all-rows +/// one. When at least one facet resolves, the unresolved ones simply pass every +/// row and the resolved ones do the narrowing. +pub fn narrowing_rows( + doc: &Document, + config: &FacetConfig, + inputs: &FacetInputs, +) -> AppResult>> { + let built = build_facets(doc, config, inputs)?; + if !built.iter().any(|b| b.narrows) { + return Ok(None); + } + Ok(Some(scan_matching(doc, &built)?)) +} + +/// Full-document scan collecting the absolute indices every built facet admits. +fn scan_matching(doc: &Document, built: &[BuiltFacet]) -> AppResult> { + let mut out = Vec::new(); + doc.visit_rows(0..doc.n_rows(), &mut |i, row| { + if built.iter().all(|b| b.eval.passes(i, row)) { + out.push(i); + } + Ok(true) + })?; + Ok(out) +} + +/// Convert the active facets to the standard filter-builder tree (one-way, +/// lossy). Facets with no faithful column-filter equivalent — semantic and the +/// four status facets, plus some nullability/boolean exclusions — are omitted +/// and listed in [`FacetConversion::dropped`]. +pub fn to_filter_group(doc: &Document, config: &FacetConfig) -> FacetConversion { + let mut nodes = Vec::new(); + let mut dropped = Vec::new(); + for spec in &config.facets { + if !spec.selection_is_active() { + continue; + } + let col = spec + .column_id + .as_deref() + .and_then(|id| resolve_col(doc, id)); + match convert_facet(spec, col) { + Ok(Some(node)) => nodes.push(node), + Ok(None) => {} + Err(reason) => dropped.push(DroppedFacet { + id: spec.id.clone(), + reason, + }), + } + } + FacetConversion { + filter: FilterGroup { + conjunction: Conjunction::And, + nodes, + }, + dropped, + } +} + +// =========================================================================== +// Facet building +// =========================================================================== + +struct BuiltFacet { + id: String, + kind: FacetKind, + column_id: Option, + mode: FacetMode, + eval: Eval, + /// This facet actually narrows the population: it carries an active + /// selection AND resolved (has a real evaluator). An active-but-unresolved + /// facet — a deleted column or an unavailable status input — is `false`, so + /// it is never mistaken for an all-rows constraint. + narrows: bool, +} + +enum Eval { + Text(Box), + Range(Box), + Cat(Box), + Status(Box), + /// Column/inputs missing: never filters, produces no counts. + Unresolved, +} + +impl Eval { + /// Whether the row satisfies this facet's selection. Status facets key on the + /// absolute row index; value facets read the row slice. + fn passes(&self, abs: usize, row: &[String]) -> bool { + match self { + Eval::Text(e) => e.passes(row), + Eval::Range(e) => e.passes(row), + Eval::Cat(e) => e.passes(row), + Eval::Status(e) => e.passes_abs(abs), + Eval::Unresolved => true, + } + } + + fn tally(&mut self, abs: usize, row: &[String]) { + match self { + Eval::Text(e) => e.tally(row), + Eval::Range(e) => e.tally(row), + Eval::Cat(e) => e.tally(row), + Eval::Status(e) => e.tally(abs), + Eval::Unresolved => {} + } + } +} + +fn build_facets( + doc: &Document, + config: &FacetConfig, + inputs: &FacetInputs, +) -> AppResult> { + let mut out = Vec::with_capacity(config.facets.len()); + for spec in &config.facets { + let mode = spec.selection.mode; + let eval = build_eval(doc, spec, inputs)?; + let narrows = spec.selection_is_active() && !matches!(eval, Eval::Unresolved); + out.push(BuiltFacet { + id: spec.id.clone(), + kind: spec.kind, + column_id: spec.column_id.clone(), + mode, + eval, + narrows, + }); + } + Ok(out) +} + +fn build_eval(doc: &Document, spec: &FacetSpec, inputs: &FacetInputs) -> AppResult { + if spec.kind.is_column_scoped() { + let Some(col) = spec + .column_id + .as_deref() + .and_then(|id| resolve_col(doc, id)) + else { + return Ok(Eval::Unresolved); + }; + let schema = doc.column_schema_at(col).cloned(); + return build_column_eval(spec, col, schema); + } + // Status facet. + match inputs.for_kind(spec.kind) { + Some(input) => Ok(Eval::Status(Box::new(StatusEval::build(spec, input)))), + None => Ok(Eval::Unresolved), + } +} + +fn build_column_eval( + spec: &FacetSpec, + col: usize, + schema: Option, +) -> AppResult { + let sel = &spec.selection; + match spec.kind { + FacetKind::Text => Ok(Eval::Text(Box::new(TextEval::build(spec, col)))), + FacetKind::Number => Ok(Eval::Range(Box::new(RangeEval::build( + spec, col, schema, false, + )?))), + FacetKind::Date => Ok(Eval::Range(Box::new(RangeEval::build( + spec, col, schema, true, + )?))), + FacetKind::Boolean => Ok(Eval::Cat(Box::new(CatEval::boolean(sel, col, schema)))), + FacetKind::Nullability => Ok(Eval::Cat(Box::new(CatEval::nullability(sel, col, schema)))), + FacetKind::Semantic => { + let Some(sem) = spec.semantic else { + return Ok(Eval::Unresolved); + }; + Ok(Eval::Cat(Box::new(CatEval::semantic(sel, col, sem)))) + } + _ => Ok(Eval::Unresolved), + } +} + +impl BuiltFacet { + fn finish(self, scan_sampled: bool, inputs_sampled: bool) -> FacetResult { + // Every facet's counts are tallied over the same `0..scan_end` window, so + // a truncated scan under-counts the four status facets exactly as it does + // value facets — flag both on `scan_sampled`. `inputs_sampled` + // additionally marks a status facet whose underlying report was itself + // only a sample. + let sampled = scan_sampled || inputs_sampled; + let base = FacetResult { + id: self.id, + kind: self.kind, + column_id: self.column_id, + mode: self.mode, + active: false, + unresolved: false, + sampled, + truncated: false, + distinct: None, + buckets: Vec::new(), + range: None, + }; + match self.eval { + Eval::Text(e) => e.finish(base), + Eval::Range(e) => e.finish(base), + Eval::Cat(e) => e.finish(base), + Eval::Status(e) => e.finish(base), + Eval::Unresolved => FacetResult { + unresolved: true, + ..base + }, + } + } +} + +// =========================================================================== +// Text facet +// =========================================================================== + +struct TextEval { + col: usize, + mode: FacetMode, + selected: HashSet, + active: bool, + top_n: usize, + search: Option, + counts: HashMap, + truncated: bool, +} + +impl TextEval { + fn build(spec: &FacetSpec, col: usize) -> TextEval { + let selected: HashSet = spec.selection.values.iter().cloned().collect(); + TextEval { + col, + mode: spec.selection.mode, + active: !selected.is_empty(), + selected, + top_n: spec.top_n.unwrap_or(DEFAULT_TEXT_TOP_N), + search: blankless(&spec.search).map(|s| s.to_lowercase()), + counts: HashMap::new(), + truncated: false, + } + } + + fn passes(&self, row: &[String]) -> bool { + if !self.active { + return true; + } + let matched = self.selected.contains(cell(row, self.col)); + match self.mode { + FacetMode::Include => matched, + FacetMode::Exclude => !matched, + } + } + + fn tally(&mut self, row: &[String]) { + let v = cell(row, self.col); + if let Some(c) = self.counts.get_mut(v) { + *c += 1; + } else if self.counts.len() < MAX_TEXT_VALUES { + self.counts.insert(v.to_string(), 1); + } else { + self.truncated = true; + } + } + + fn finish(self, base: FacetResult) -> FacetResult { + let distinct = self.counts.len() as u64; + let mut buckets: Vec = Vec::new(); + let mut placed: HashSet<&str> = HashSet::new(); + + // Selected values are always shown (checked), even at count 0 — but bound + // the emitted buckets by MAX_TEXT_VALUES so a pathological or hand-edited + // saved view with a huge `selection.values` array can never produce an + // unbounded payload (the "never a full dump" guarantee). Every selected + // key is still marked `placed` (so it is excluded from `rest`) and still + // filters exactly via `self.selected` in `passes`; only the DISPLAY list + // is capped. The cap is deterministic (sorted, leading MAX_TEXT_VALUES). + let mut sorted_selected: Vec<&String> = self.selected.iter().collect(); + sorted_selected.sort(); + for (i, key) in sorted_selected.iter().enumerate() { + placed.insert(key.as_str()); + if i < MAX_TEXT_VALUES { + let count = self.counts.get(*key).copied().unwrap_or(0); + buckets.push(text_bucket(key, count, true)); + } + } + + // Then the top values by count among the (search-narrowed) remainder. + let mut rest: Vec<(&String, u64)> = self + .counts + .iter() + .filter(|(k, _)| !placed.contains(k.as_str())) + .filter(|(k, _)| match &self.search { + Some(s) => k.to_lowercase().contains(s.as_str()), + None => true, + }) + .map(|(k, v)| (k, *v)) + .collect(); + rest.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(b.0))); + for (key, count) in rest.into_iter().take(self.top_n) { + buckets.push(text_bucket(key, count, false)); + } + + FacetResult { + active: self.active, + truncated: self.truncated, + distinct: Some(distinct), + buckets, + ..base + } + } +} + +fn text_bucket(key: &str, count: u64, selected: bool) -> FacetBucket { + FacetBucket { + key: key.to_string(), + label: key.to_string(), + count, + selected, + lo: None, + hi: None, + } +} + +// =========================================================================== +// Number / date facet (histogram + continuous range) +// =========================================================================== + +struct RangeEval { + col: usize, + date: bool, + schema: Option, + mode: FacetMode, + active: bool, + min: Option, + max: Option, + sel_min: Option, + sel_max: Option, + nbins: usize, + obs_min: Option, + obs_max: Option, + edges: Vec, + counts: Vec, +} + +impl RangeEval { + fn build( + spec: &FacetSpec, + col: usize, + schema: Option, + date: bool, + ) -> AppResult { + let range = &spec.selection.range; + let sel_min = blankless(&range.min).map(str::to_string); + let sel_max = blankless(&range.max).map(str::to_string); + let parse = |s: &Option| -> AppResult> { + match s { + Some(v) => Ok(Some(parse_bound(v, schema.as_ref(), date)?)), + None => Ok(None), + } + }; + let min = parse(&sel_min)?; + let max = parse(&sel_max)?; + Ok(RangeEval { + col, + date, + schema, + mode: spec.selection.mode, + active: min.is_some() || max.is_some(), + min, + max, + sel_min, + sel_max, + nbins: spec.bins.unwrap_or(DEFAULT_BINS).clamp(1, MAX_BINS), + obs_min: None, + obs_max: None, + edges: Vec::new(), + counts: Vec::new(), + }) + } + + fn value(&self, s: &str) -> Option { + if self.date { + date_value(s, self.schema.as_ref()) + } else { + numeric_value(s, self.schema.as_ref()) + } + } + + fn in_range(&self, v: f64) -> bool { + self.min.is_none_or(|lo| v >= lo) && self.max.is_none_or(|hi| v <= hi) + } + + fn passes(&self, row: &[String]) -> bool { + if !self.active { + return true; + } + let matched = self + .value(cell(row, self.col)) + .is_some_and(|v| self.in_range(v)); + match self.mode { + FacetMode::Include => matched, + FacetMode::Exclude => !matched, + } + } + + fn observe(&mut self, row: &[String]) { + if let Some(v) = self.value(cell(row, self.col)) { + self.obs_min = Some(self.obs_min.map_or(v, |m| m.min(v))); + self.obs_max = Some(self.obs_max.map_or(v, |m| m.max(v))); + } + } + + fn finalize_bins(&mut self) { + let (Some(lo), Some(hi)) = (self.obs_min, self.obs_max) else { + return; + }; + if lo >= hi { + self.edges = vec![lo, hi]; + self.counts = vec![0]; + return; + } + let n = self.nbins.max(1); + let step = (hi - lo) / n as f64; + let mut edges: Vec = (0..=n).map(|i| lo + step * i as f64).collect(); + *edges.last_mut().unwrap() = hi; // pin the top edge exactly + self.counts = vec![0; n]; + self.edges = edges; + } + + fn bin_of(&self, v: f64) -> usize { + let n = self.counts.len(); + if n == 0 { + return 0; + } + if v <= self.edges[0] { + return 0; + } + if v >= self.edges[n] { + return n - 1; + } + // Linear scan (n <= MAX_BINS): the bin whose upper edge first exceeds v. + for (i, w) in self.edges.windows(2).enumerate() { + if v < w[1] { + return i; + } + } + n - 1 + } + + fn tally(&mut self, row: &[String]) { + if self.counts.is_empty() { + return; + } + if let Some(v) = self.value(cell(row, self.col)) { + let bin = self.bin_of(v); + self.counts[bin] += 1; + } + } + + /// Format a numeric edge or a date-timestamp edge for display. + fn fmt_edge(&self, v: f64) -> String { + if self.date { + fmt_ts(v as i64, is_datetime(self.schema.as_ref())) + } else { + fmt_num(v) + } + } + + fn finish(self, base: FacetResult) -> FacetResult { + let mut buckets = Vec::with_capacity(self.counts.len()); + for (i, &count) in self.counts.iter().enumerate() { + let lo = self.edges[i]; + let hi = self.edges[i + 1]; + buckets.push(FacetBucket { + key: format!("b{i}"), + label: format!("{} – {}", self.fmt_edge(lo), self.fmt_edge(hi)), + count, + selected: false, + lo: Some(lo), + hi: Some(hi), + }); + } + let range = RangeInfo { + min: self.obs_min.map(|v| self.fmt_edge(v)), + max: self.obs_max.map(|v| self.fmt_edge(v)), + selected_min: self.sel_min, + selected_max: self.sel_max, + }; + FacetResult { + active: self.active, + buckets, + range: Some(range), + ..base + } + } +} + +// =========================================================================== +// Categorical facet (boolean / nullability / semantic — one category per row) +// =========================================================================== + +struct CatEval { + col: usize, + classifier: CatClassifier, + categories: Vec, + selected: u64, + active: bool, + mode: FacetMode, + counts: Vec, +} + +struct CatDef { + key: &'static str, + label: String, +} + +enum CatClassifier { + Boolean(Option), + Nullability(ColumnSchema), + Semantic(SemanticType), +} + +impl CatEval { + fn new( + col: usize, + classifier: CatClassifier, + categories: Vec, + sel: &FacetSelection, + ) -> CatEval { + let chosen = sel.value_set(); + let mut selected = 0u64; + for (i, c) in categories.iter().enumerate() { + if chosen.contains(c.key) { + selected |= 1 << i; + } + } + let counts = vec![0; categories.len()]; + CatEval { + col, + classifier, + categories, + active: selected != 0, + selected, + mode: sel.mode, + counts, + } + } + + fn boolean(sel: &FacetSelection, col: usize, schema: Option) -> CatEval { + let cats = vec![ + CatDef { + key: "true", + label: "True".into(), + }, + CatDef { + key: "false", + label: "False".into(), + }, + CatDef { + key: "blank", + label: "Blank".into(), + }, + CatDef { + key: "other", + label: "Other".into(), + }, + ]; + CatEval::new(col, CatClassifier::Boolean(schema), cats, sel) + } + + fn nullability(sel: &FacetSelection, col: usize, schema: Option) -> CatEval { + // Without a declared schema, classify against permissive text so blanks + // still separate from values (null-token / invalid never arise). + let schema = + schema.unwrap_or_else(|| ColumnSchema::new("", "", crate::schema::LogicalType::Text)); + let cats = vec![ + CatDef { + key: "value", + label: "Has value".into(), + }, + CatDef { + key: "blank", + label: "Blank".into(), + }, + CatDef { + key: "null", + label: "Null token".into(), + }, + CatDef { + key: "invalid", + label: "Invalid".into(), + }, + ]; + CatEval::new(col, CatClassifier::Nullability(schema), cats, sel) + } + + fn semantic(sel: &FacetSelection, col: usize, sem: SemanticType) -> CatEval { + let cats = vec![ + CatDef { + key: "match", + label: "Matches".into(), + }, + CatDef { + key: "mismatch", + label: "Doesn't match".into(), + }, + CatDef { + key: "blank", + label: "Blank".into(), + }, + ]; + CatEval::new(col, CatClassifier::Semantic(sem), cats, sel) + } + + fn classify(&self, s: &str) -> usize { + match &self.classifier { + CatClassifier::Boolean(schema) => bool_bucket(s, schema.as_ref()), + CatClassifier::Nullability(schema) => match schema::classify(Some(s), schema) { + CellState::Valid(_) => 0, + CellState::Empty | CellState::Missing => 1, + CellState::NullToken => 2, + CellState::Invalid(_) => 3, + }, + CatClassifier::Semantic(sem) => { + if s.trim().is_empty() { + 2 + } else if semantic::matches_type(s, *sem) { + 0 + } else { + 1 + } + } + } + } + + fn passes(&self, row: &[String]) -> bool { + if !self.active { + return true; + } + let cat = self.classify(cell(row, self.col)); + let matched = (self.selected >> cat) & 1 == 1; + match self.mode { + FacetMode::Include => matched, + FacetMode::Exclude => !matched, + } + } + + fn tally(&mut self, row: &[String]) { + let cat = self.classify(cell(row, self.col)); + self.counts[cat] += 1; + } + + fn finish(self, base: FacetResult) -> FacetResult { + let buckets = self + .categories + .iter() + .enumerate() + .map(|(i, c)| FacetBucket { + key: c.key.to_string(), + label: c.label.clone(), + count: self.counts[i], + selected: (self.selected >> i) & 1 == 1, + lo: None, + hi: None, + }) + .collect(); + FacetResult { + active: self.active, + buckets, + ..base + } + } +} + +// =========================================================================== +// Status facet (diagnostics / validation / duplicate / annotation) +// =========================================================================== + +struct StatusEval { + row_mask: HashMap, + categories: Vec, + none_label: Option, + selected: u64, + none_selected: bool, + active: bool, + mode: FacetMode, + counts: Vec, + none_count: u64, +} + +impl StatusEval { + fn build(spec: &FacetSpec, input: &StatusInput) -> StatusEval { + let categories: Vec = input + .categories + .iter() + .take(MAX_STATUS_CATEGORIES) + .cloned() + .collect(); + let mut row_mask: HashMap = HashMap::new(); + for (i, cat) in categories.iter().enumerate() { + let bit = 1u64 << i; + for &row in &cat.rows { + *row_mask.entry(row).or_insert(0) |= bit; + } + } + let chosen = spec.selection.value_set(); + let mut selected = 0u64; + for (i, cat) in categories.iter().enumerate() { + if chosen.contains(cat.key.as_str()) { + selected |= 1 << i; + } + } + let none_selected = input.none_label.is_some() && chosen.contains(NONE_KEY); + StatusEval { + row_mask, + counts: vec![0; categories.len()], + categories, + none_label: input.none_label.clone(), + active: selected != 0 || none_selected, + selected, + none_selected, + mode: spec.selection.mode, + none_count: 0, + } + } + + fn mask(&self, abs: usize) -> u64 { + self.row_mask.get(&abs).copied().unwrap_or(0) + } + + fn passes_abs(&self, abs: usize) -> bool { + if !self.active { + return true; + } + let mask = self.mask(abs); + let matched = (mask & self.selected != 0) || (mask == 0 && self.none_selected); + match self.mode { + FacetMode::Include => matched, + FacetMode::Exclude => !matched, + } + } + + fn tally(&mut self, abs: usize) { + let mask = self.mask(abs); + if mask == 0 { + self.none_count += 1; + return; + } + for (i, c) in self.counts.iter_mut().enumerate() { + if (mask >> i) & 1 == 1 { + *c += 1; + } + } + } + + fn finish(self, base: FacetResult) -> FacetResult { + let mut buckets: Vec = self + .categories + .iter() + .enumerate() + .map(|(i, c)| FacetBucket { + key: c.key.clone(), + label: c.label.clone(), + count: self.counts[i], + selected: (self.selected >> i) & 1 == 1, + lo: None, + hi: None, + }) + .collect(); + if let Some(label) = &self.none_label { + buckets.push(FacetBucket { + key: NONE_KEY.to_string(), + label: label.clone(), + count: self.none_count, + selected: self.none_selected, + lo: None, + hi: None, + }); + } + FacetResult { + active: self.active, + buckets, + ..base + } + } +} + +/// Reserved selection key for a status facet's synthesized "none" bucket. +const NONE_KEY: &str = "__none__"; + +// =========================================================================== +// facets -> filter-builder conversion +// =========================================================================== + +fn convert_facet(spec: &FacetSpec, col: Option) -> Result, String> { + let sel = &spec.selection; + let include = sel.mode == FacetMode::Include; + match spec.kind { + FacetKind::Text => { + let col = col.ok_or_else(|| "column not found".to_string())?; + let (op, conj) = if include { + (FilterOp::Equals, Conjunction::Or) + } else { + (FilterOp::NotEquals, Conjunction::And) + }; + let nodes = sel + .values + .iter() + .map(|v| condition(col, op, v, true)) + .collect(); + Ok(Some(FilterNode::Group(FilterGroup { + conjunction: conj, + nodes, + }))) + } + FacetKind::Number | FacetKind::Date => { + let col = col.ok_or_else(|| "column not found".to_string())?; + let min = blankless(&sel.range.min); + let max = blankless(&sel.range.max); + if include { + let mut nodes = Vec::new(); + if let Some(lo) = min { + nodes.push(condition(col, FilterOp::Gte, lo, false)); + } + if let Some(hi) = max { + nodes.push(condition(col, FilterOp::Lte, hi, false)); + } + Ok(Some(FilterNode::Group(FilterGroup { + conjunction: Conjunction::And, + nodes, + }))) + } else { + // Exclude keeps every row the include predicate rejects: not just + // the parseable out-of-range cells, but also the blank/null/invalid + // ones (the facet negates `parses && in range`). A `< min OR > max` + // filter can only ever match parseable, out-of-range cells — the + // filter engine's typed comparisons never match blank/invalid — so + // emitting it would silently drop the blank/unparseable rows the + // facet keeps. There is no filter op for "not a valid number/date", + // so report the exclusion as lossy instead of inverting semantics. + Err( + "exclude-mode number/date ranges have no exact filter equivalent \ + (they also keep blank and unparseable cells)" + .into(), + ) + } + } + FacetKind::Nullability => { + if !include { + return Err("exclude-mode nullability has no filter equivalent".into()); + } + let col = col.ok_or_else(|| "column not found".to_string())?; + let chosen = sel.value_set(); + // Only the cleanly expressible single-bucket selections convert. + if chosen.len() == 1 && chosen.contains("blank") { + Ok(Some(FilterNode::Condition(FilterCondition { + column: col, + op: FilterOp::IsEmpty, + value: String::new(), + case_sensitive: false, + }))) + } else if chosen.len() == 1 && chosen.contains("value") { + Ok(Some(FilterNode::Condition(FilterCondition { + column: col, + op: FilterOp::NotEmpty, + value: String::new(), + case_sensitive: false, + }))) + } else { + Err("this nullability selection has no exact filter equivalent".into()) + } + } + FacetKind::Boolean => { + Err("boolean facets don't convert to a filter (multiple spellings)".into()) + } + FacetKind::Semantic => Err("semantic facets have no column-filter equivalent".into()), + FacetKind::Diagnostics + | FacetKind::Validation + | FacetKind::Duplicate + | FacetKind::Annotation => Err("status facets have no column-filter equivalent".into()), + } +} + +fn condition(column: usize, op: FilterOp, value: &str, case_sensitive: bool) -> FilterNode { + FilterNode::Condition(FilterCondition { + column, + op, + value: value.to_string(), + case_sensitive, + }) +} + +// =========================================================================== +// Shared value helpers +// =========================================================================== + +fn cell(row: &[String], col: usize) -> &str { + row.get(col).map(String::as_str).unwrap_or("") +} + +fn resolve_col(doc: &Document, column_id: &str) -> Option { + doc.column_ids().iter().position(|id| id == column_id) +} + +/// `Some(trimmed)` unless the option is absent or blank. +fn blankless(s: &Option) -> Option<&str> { + match s { + Some(v) if !v.trim().is_empty() => Some(v.trim()), + _ => None, + } +} + +fn numeric_value(s: &str, schema: Option<&ColumnSchema>) -> Option { + if let Some(sc) = schema { + if sc.logical_type.is_numeric() { + return match schema::numeric_cell(sc, s) { + NumericCell::Value(v) => Some(v), + _ => None, + }; + } + } + analyze::as_number(s) +} + +fn date_value(s: &str, schema: Option<&ColumnSchema>) -> Option { + if s.trim().is_empty() { + return None; + } + if let Some(sc) = schema { + if sc.logical_type.is_temporal() { + return match schema::classify(Some(s), sc) { + CellState::Valid(TypedValue::Date(d)) => { + Some(d.and_hms_opt(0, 0, 0)?.and_utc().timestamp() as f64) + } + CellState::Valid(TypedValue::DateTime(dt)) => Some(dt.and_utc().timestamp() as f64), + _ => None, + }; + } + } + analyze::parse_date(s).map(|dt| dt.and_utc().timestamp() as f64) +} + +fn parse_bound(s: &str, schema: Option<&ColumnSchema>, date: bool) -> AppResult { + let parsed = if date { + date_value(s, schema) + } else { + numeric_value(s, schema) + }; + parsed.ok_or_else(|| { + AppError::invalid(format!( + "'{s}' is not a valid {} bound", + if date { "date" } else { "numeric" } + )) + }) +} + +fn is_datetime(schema: Option<&ColumnSchema>) -> bool { + matches!( + schema.map(|s| s.logical_type), + Some(crate::schema::LogicalType::Datetime) + ) +} + +/// Boolean bucket index: 0 true, 1 false, 2 blank, 3 other. Uses the declared +/// schema when present, else a permissive heuristic (numeric flags count). +fn bool_bucket(s: &str, schema: Option<&ColumnSchema>) -> usize { + if let Some(sc) = schema { + if sc.logical_type == crate::schema::LogicalType::Boolean { + return match schema::classify(Some(s), sc) { + CellState::Valid(TypedValue::Boolean(true)) => 0, + CellState::Valid(TypedValue::Boolean(false)) => 1, + CellState::Empty | CellState::Missing | CellState::NullToken => 2, + _ => 3, + }; + } + } + let t = s.trim(); + if t.is_empty() { + return 2; + } + match t.to_ascii_lowercase().as_str() { + "true" | "yes" | "1" | "t" | "y" => 0, + "false" | "no" | "0" | "f" | "n" => 1, + _ => 3, + } +} + +fn fmt_num(v: f64) -> String { + if v.is_finite() && v.fract() == 0.0 && v.abs() < 1e15 { + (v as i64).to_string() + } else { + let s = format!("{v:.4}"); + s.trim_end_matches('0').trim_end_matches('.').to_string() + } +} + +fn fmt_ts(secs: i64, datetime: bool) -> String { + match chrono::DateTime::from_timestamp(secs, 0) { + Some(dt) => { + let naive = dt.naive_utc(); + if datetime { + naive.format("%Y-%m-%d %H:%M:%S").to_string() + } else { + naive.format("%Y-%m-%d").to_string() + } + } + None => secs.to_string(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::annotations::{AnnotationStore, RowMarkPatch}; + use crate::parse::{parse, ParseSettings}; + use crate::schema::LogicalType; + use crate::tabular::DocumentSource; + + fn doc(csv: &str) -> Document { + let parsed = parse(csv.as_bytes(), &ParseSettings::default()).unwrap(); + Document::from_parsed(1, None, parsed, true) + } + + fn spec(id: &str, kind: FacetKind, column_id: &str) -> FacetSpec { + FacetSpec { + id: id.into(), + kind, + column_id: Some(column_id.into()), + semantic: None, + selection: FacetSelection::default(), + top_n: None, + search: None, + bins: None, + pinned: false, + collapsed: false, + width: None, + } + } + + fn include(values: &[&str]) -> FacetSelection { + FacetSelection { + mode: FacetMode::Include, + values: values.iter().map(|s| s.to_string()).collect(), + range: FacetRange::default(), + } + } + + fn find<'a>(rs: &'a FacetResultSet, id: &str) -> &'a FacetResult { + rs.facets + .iter() + .find(|f| f.id == id) + .expect("facet present") + } + + fn count(f: &FacetResult, key: &str) -> u64 { + f.buckets + .iter() + .find(|b| b.key == key) + .map(|b| b.count) + .unwrap_or_else(|| panic!("bucket {key} present")) + } + + // city,age,active + // 0 NYC,20,true + // 1 NYC,30,false + // 2 LA,20,true + // 3 LA,40,true + // 4 NYC,30,true + fn sample_doc() -> Document { + doc("city,age,active\nNYC,20,true\nNYC,30,false\nLA,20,true\nLA,40,true\nNYC,30,true\n") + } + + fn config_city_age_bool() -> FacetConfig { + let mut age = spec("age", FacetKind::Number, "c1"); + age.bins = Some(2); // edges [20,30,40]: b0=[20,30) b1=[30,40] + FacetConfig { + facets: vec![ + spec("city", FacetKind::Text, "c0"), + age, + spec("active", FacetKind::Boolean, "c2"), + ], + } + } + + #[test] + fn unfiltered_counts_are_full_population() { + let d = sample_doc(); + let cfg = config_city_age_bool(); + let rs = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + assert_eq!(rs.matched_rows, 5); + assert_eq!(rs.total_rows, 5); + assert!(!rs.sampled); + + let city = find(&rs, "city"); + assert_eq!(count(city, "NYC"), 3); + assert_eq!(count(city, "LA"), 2); + assert_eq!(city.distinct, Some(2)); + + // age b0=[20,30) -> rows 0,2 ; b1=[30,40] -> rows 1,3,4 + let age = find(&rs, "age"); + assert_eq!(count(age, "b0"), 2); + assert_eq!(count(age, "b1"), 3); + + let active = find(&rs, "active"); + assert_eq!(count(active, "true"), 4); + assert_eq!(count(active, "false"), 1); + } + + #[test] + fn cross_filter_counts_reflect_other_facets() { + let d = sample_doc(); + let mut cfg = config_city_age_bool(); + cfg.facets[0].selection = include(&["NYC"]); // city = NYC {0,1,4} + + let rs = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + // Only city active -> population is NYC = 3. + assert_eq!(rs.matched_rows, 3); + + // City counts reflect all OTHER facets (none active) -> full. + let city = find(&rs, "city"); + assert_eq!(count(city, "NYC"), 3); + assert_eq!(count(city, "LA"), 2); + assert!( + city.buckets + .iter() + .find(|b| b.key == "NYC") + .unwrap() + .selected + ); + + // Boolean counts over the city=NYC population {0,1,4}: true=2,false=1. + let active = find(&rs, "active"); + assert_eq!(count(active, "true"), 2); + assert_eq!(count(active, "false"), 1); + + // Age counts over {0,1,4}: ages 20,30,30 -> b0=1, b1=2. + let age = find(&rs, "age"); + assert_eq!(count(age, "b0"), 1); + assert_eq!(count(age, "b1"), 2); + } + + #[test] + fn two_active_facets_and_clear_retention() { + let d = sample_doc(); + let mut cfg = config_city_age_bool(); + cfg.facets[0].selection = include(&["NYC"]); // city=NYC {0,1,4} + cfg.facets[2].selection = include(&["true"]); // active=true {0,2,3,4} + + let rs = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + // city=NYC AND active=true -> {0,4} + assert_eq!(rs.matched_rows, 2); + + // City counts against the OTHER facets (active=true) population {0,2,3,4}: + // NYC among those = {0,4}=2, LA = {2,3}=2. + let city = find(&rs, "city"); + assert_eq!(count(city, "NYC"), 2); + assert_eq!(count(city, "LA"), 2); + + // Boolean counts against city=NYC {0,1,4}: true=2, false=1. + let active = find(&rs, "active"); + assert_eq!(count(active, "true"), 2); + assert_eq!(count(active, "false"), 1); + + // Age counts against city=NYC AND active=true = {0,4}: ages 20,30 -> b0=1,b1=1. + let age = find(&rs, "age"); + assert_eq!(count(age, "b0"), 1); + assert_eq!(count(age, "b1"), 1); + + // Clearing the boolean facet must leave city's selection intact and its + // counts revert to the full population. + cfg.facets[2].selection = FacetSelection::default(); + let rs2 = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + assert_eq!(rs2.matched_rows, 3); // just city=NYC again + let city2 = find(&rs2, "city"); + assert_eq!(count(city2, "NYC"), 3); + assert_eq!(count(city2, "LA"), 2); + } + + #[test] + fn include_exclude_is_deterministic() { + let d = sample_doc(); + let inputs = FacetInputs::default(); + + let mut inc = FacetConfig { + facets: vec![spec("city", FacetKind::Text, "c0")], + }; + inc.facets[0].selection = include(&["NYC"]); + assert_eq!(matching_rows(&d, &inc, &inputs).unwrap(), vec![0, 1, 4]); + + let mut exc = inc.clone(); + exc.facets[0].selection.mode = FacetMode::Exclude; + assert_eq!(matching_rows(&d, &exc, &inputs).unwrap(), vec![2, 3]); + } + + #[test] + fn numeric_range_selection_applies_exactly() { + let d = sample_doc(); + let inputs = FacetInputs::default(); + let mut cfg = FacetConfig { + facets: vec![spec("age", FacetKind::Number, "c1")], + }; + cfg.facets[0].selection = FacetSelection { + mode: FacetMode::Include, + values: vec![], + range: FacetRange { + min: Some("25".into()), + max: None, + }, + }; + // age >= 25 -> rows 1(30),3(40),4(30) + assert_eq!(matching_rows(&d, &cfg, &inputs).unwrap(), vec![1, 3, 4]); + } + + #[test] + fn matching_rows_matches_all_when_inactive() { + let d = sample_doc(); + let cfg = config_city_age_bool(); + assert_eq!( + matching_rows(&d, &cfg, &FacetInputs::default()).unwrap(), + vec![0, 1, 2, 3, 4] + ); + } + + #[test] + fn compute_and_apply_are_non_destructive() { + let mut d = sample_doc(); + let rev = d.revision(); + let mut cfg = config_city_age_bool(); + cfg.facets[0].selection = include(&["NYC"]); + + // Computing counts touches nothing. + let _ = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + assert_eq!(d.revision(), rev); + assert!(!d.is_dirty()); + + // Applying facets sets the row view like a filter — never dirties. + let rows = matching_rows(&d, &cfg, &FacetInputs::default()).unwrap(); + d.set_filter(rows).unwrap(); + assert!(!d.is_dirty()); + assert_eq!(d.visible_len(), 3); + } + + #[test] + fn text_facet_is_bounded_and_truncates() { + // 12_000 distinct values trips the MAX_TEXT_VALUES cap. + let mut csv = String::from("id\n"); + for i in 0..12_000 { + csv.push_str(&format!("v{i}\n")); + } + let d = doc(&csv); + let mut s = spec("ids", FacetKind::Text, "c0"); + s.top_n = Some(5); + let cfg = FacetConfig { facets: vec![s] }; + let rs = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + let f = find(&rs, "ids"); + assert!(f.truncated, "distinct beyond the cap must flag truncated"); + assert_eq!(f.distinct, Some(MAX_TEXT_VALUES as u64)); + assert!( + f.buckets.len() <= 5, + "bounded DTO: at most top_n buckets, got {}", + f.buckets.len() + ); + } + + #[test] + fn text_search_narrows_returned_values() { + let d = doc("name\napple\napricot\nbanana\ncherry\n"); + let mut s = spec("name", FacetKind::Text, "c0"); + s.search = Some("ap".into()); + let cfg = FacetConfig { facets: vec![s] }; + let rs = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + let f = find(&rs, "name"); + let keys: HashSet<&str> = f.buckets.iter().map(|b| b.key.as_str()).collect(); + assert_eq!(keys, HashSet::from(["apple", "apricot"])); + assert_eq!(f.distinct, Some(4)); // distinct still counts the whole column + } + + #[test] + fn nullability_distinguishes_blank_null_invalid_value() { + // Declared integer column with a NULL token; row 2 blank, row 3 invalid. + let mut d = doc("n,k\n10,a\nNULL,b\n,c\nxx,d\n"); + let mut schema = ColumnSchema::new(d.column_ids()[0].clone(), "n", LogicalType::Integer); + schema.null_tokens = vec!["NULL".into()]; + d.set_column_schema(schema); + + let cfg = FacetConfig { + facets: vec![spec("null", FacetKind::Nullability, "c0")], + }; + let rs = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + let f = find(&rs, "null"); + assert_eq!(count(f, "value"), 1); // "10" + assert_eq!(count(f, "null"), 1); // "NULL" + assert_eq!(count(f, "blank"), 1); // "" + assert_eq!(count(f, "invalid"), 1); // "xx" + } + + #[test] + fn view_config_round_trips_through_json() { + let mut age = spec("age", FacetKind::Number, "c1"); + age.bins = Some(8); + age.selection = FacetSelection { + mode: FacetMode::Include, + values: vec![], + range: FacetRange { + min: Some("18".into()), + max: Some("65".into()), + }, + }; + age.pinned = true; + age.width = Some(240.0); + let mut city = spec("city", FacetKind::Text, "c0"); + city.selection = include(&["NYC", "LA"]); + city.selection.mode = FacetMode::Exclude; + city.top_n = Some(30); + let cfg = FacetConfig { + facets: vec![city, age], + }; + let json = serde_json::to_string(&cfg).unwrap(); + let back: FacetConfig = serde_json::from_str(&json).unwrap(); + assert_eq!(cfg, back); + } + + #[test] + fn conversion_matches_the_facet_filter_for_convertible_facets() { + let d = sample_doc(); + let inputs = FacetInputs::default(); + + // Text include + numeric range: both should convert to a filter that + // selects exactly the same rows the facet engine would. + let mut cfg = FacetConfig { + facets: vec![ + spec("city", FacetKind::Text, "c0"), + spec("age", FacetKind::Number, "c1"), + ], + }; + cfg.facets[0].selection = include(&["NYC"]); + cfg.facets[1].selection = FacetSelection { + mode: FacetMode::Include, + values: vec![], + range: FacetRange { + min: Some("25".into()), + max: None, + }, + }; + let conv = to_filter_group(&d, &cfg); + assert!(conv.dropped.is_empty()); + let via_filter = crate::filter::matching_rows(&d, &conv.filter).unwrap(); + let via_facets = matching_rows(&d, &cfg, &inputs).unwrap(); + assert_eq!(via_filter, via_facets); + // city=NYC AND age>=25 -> rows 1,4 + assert_eq!(via_facets, vec![1, 4]); + } + + #[test] + fn conversion_reports_dropped_status_and_semantic_facets() { + let d = sample_doc(); + let mut sem = spec("sem", FacetKind::Semantic, "c0"); + sem.semantic = Some(SemanticType::Email); + sem.selection = include(&["match"]); + let mut ann = FacetSpec { + id: "ann".into(), + kind: FacetKind::Annotation, + column_id: None, + semantic: None, + selection: include(&["starred"]), + top_n: None, + search: None, + bins: None, + pinned: false, + collapsed: false, + width: None, + }; + // exclude-mode to also exercise that path is not required; keep include. + ann.selection.mode = FacetMode::Include; + let cfg = FacetConfig { + facets: vec![sem, ann], + }; + let conv = to_filter_group(&d, &cfg); + let ids: HashSet<&str> = conv.dropped.iter().map(|d| d.id.as_str()).collect(); + assert!(ids.contains("sem")); + assert!(ids.contains("ann")); + assert!(conv.filter.nodes.is_empty()); + } + + #[test] + fn exclude_range_is_reported_lossy_not_inverted() { + // Number column with a blank cell (row 1); the second column keeps the + // blank line from being skipped as an all-empty row. + let d = doc("n,k\n10,a\n,b\n40,c\n"); + let mut s = spec("n", FacetKind::Number, "c0"); + s.selection = FacetSelection { + mode: FacetMode::Exclude, + values: vec![], + range: FacetRange { + min: Some("20".into()), + max: None, + }, + }; + let cfg = FacetConfig { facets: vec![s] }; + + // The exclude facet keeps every row that is NOT (parseable AND >= 20): + // the below-range 10 (row 0) AND the blank cell (row 1). 40 is dropped. + assert_eq!( + matching_rows(&d, &cfg, &FacetInputs::default()).unwrap(), + vec![0, 1] + ); + + // Converting must NOT emit a `< 20` filter — that would silently drop the + // blank row the facet keeps (typed comparisons never match blank/invalid). + // It reports the facet as lossy/dropped instead of inverting semantics. + let conv = to_filter_group(&d, &cfg); + assert!( + conv.filter.nodes.is_empty(), + "no inverted range filter is emitted for an exclude range" + ); + let ids: HashSet<&str> = conv.dropped.iter().map(|d| d.id.as_str()).collect(); + assert!( + ids.contains("n"), + "the exclude range is reported as dropped" + ); + } + + #[test] + fn nullability_blank_converts_to_is_empty() { + // The second column keeps the blank-cell row from being an all-empty + // line, which the parser would skip entirely. + let d = doc("a,b\nx,1\n,2\ny,3\n"); + let mut s = spec("nb", FacetKind::Nullability, "c0"); + s.selection = include(&["blank"]); + let cfg = FacetConfig { facets: vec![s] }; + let conv = to_filter_group(&d, &cfg); + assert!(conv.dropped.is_empty()); + let via_filter = crate::filter::matching_rows(&d, &conv.filter).unwrap(); + let via_facets = matching_rows(&d, &cfg, &FacetInputs::default()).unwrap(); + assert_eq!(via_filter, via_facets); + assert_eq!(via_facets, vec![1]); // only the genuinely blank row + } + + #[test] + fn unresolved_column_facet_never_filters() { + let d = sample_doc(); + let cfg = FacetConfig { + facets: vec![spec("gone", FacetKind::Text, "c99")], // no such column id + }; + let rs = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + let f = find(&rs, "gone"); + assert!(f.unresolved); + assert!(!f.active); + assert_eq!(rs.matched_rows, 5); + assert_eq!( + matching_rows(&d, &cfg, &FacetInputs::default()).unwrap(), + vec![0, 1, 2, 3, 4] + ); + } + + #[test] + fn narrowing_rows_skips_apply_when_only_unresolved_facets_are_active() { + let d = sample_doc(); + let mut cfg = FacetConfig { + facets: vec![spec("gone", FacetKind::Text, "c99")], // deleted column + }; + cfg.facets[0].selection = include(&["NYC"]); // active, but unresolved + + // `any_active` is true, yet nothing resolves to a real constraint, so + // apply must be a no-op (None) rather than an all-rows filter that would + // falsely mark the document filtered. + assert!(cfg.any_active()); + assert_eq!( + narrowing_rows(&d, &cfg, &FacetInputs::default()).unwrap(), + None + ); + } + + #[test] + fn narrowing_rows_applies_resolved_and_ignores_unresolved() { + let d = sample_doc(); + let mut cfg = FacetConfig { + facets: vec![ + spec("city", FacetKind::Text, "c0"), // resolved -> narrows + spec("gone", FacetKind::Text, "c99"), // active but unresolved + ], + }; + cfg.facets[0].selection = include(&["NYC"]); // city=NYC -> {0,1,4} + cfg.facets[1].selection = include(&["x"]); // unresolved: passes every row + + // The resolved facet narrows; the unresolved one is ignored (not treated + // as an all-rows constraint that would wipe out the narrowing). + assert_eq!( + narrowing_rows(&d, &cfg, &FacetInputs::default()).unwrap(), + Some(vec![0, 1, 4]) + ); + } + + // ----- annotation facet integration (F40, wired for real) -------------- + + fn annotated_doc() -> (Document, FacetInputs) { + // 5 rows so one row carries no annotation. + let d = doc("id,name\n1,Ada\n2,Bob\n3,Cy\n4,Di\n5,Ed\n"); + let mut store = AnnotationStore::default(); + let s = DocumentSource::new(&d); + // Row 0: star + tag keep. Row 1: flag. Row 2: star. Row 3: tag keep. + store + .edit_row_marks( + &s, + 0, + &RowMarkPatch { + star: Some(true), + add_tags: vec!["keep".into()], + ..Default::default() + }, + None, + ) + .unwrap(); + store + .edit_row_marks( + &s, + 1, + &RowMarkPatch { + flag: Some(true), + ..Default::default() + }, + None, + ) + .unwrap(); + store + .edit_row_marks( + &s, + 2, + &RowMarkPatch { + star: Some(true), + ..Default::default() + }, + None, + ) + .unwrap(); + store + .edit_row_marks( + &s, + 3, + &RowMarkPatch { + add_tags: vec!["keep".into()], + ..Default::default() + }, + None, + ) + .unwrap(); + let index = store.mark_index(&s, None).unwrap(); + let inputs = FacetInputs { + annotation: Some(StatusInput::from_marks(&index)), + ..Default::default() + }; + (d, inputs) + } + + #[test] + fn annotation_facet_counts_marks() { + let (d, inputs) = annotated_doc(); + let cfg = FacetConfig { + facets: vec![FacetSpec { + id: "ann".into(), + kind: FacetKind::Annotation, + column_id: None, + semantic: None, + selection: FacetSelection::default(), + top_n: None, + search: None, + bins: None, + pinned: false, + collapsed: false, + width: None, + }], + }; + let rs = compute(&d, &cfg, &inputs, None).unwrap(); + let f = find(&rs, "ann"); + assert_eq!(count(f, "starred"), 2); // rows 0,2 + assert_eq!(count(f, "flagged"), 1); // row 1 + assert_eq!(count(f, "tag:keep"), 2); // rows 0,3 + assert_eq!(count(f, NONE_KEY), 1); // row 4 carries nothing + } + + #[test] + fn annotation_facet_filters_and_cross_filters() { + let (d, inputs) = annotated_doc(); + let mut ann = FacetSpec { + id: "ann".into(), + kind: FacetKind::Annotation, + column_id: None, + semantic: None, + selection: include(&["starred"]), + top_n: None, + search: None, + bins: None, + pinned: false, + collapsed: false, + width: None, + }; + ann.selection.mode = FacetMode::Include; + let cfg = FacetConfig { + facets: vec![ann, spec("name", FacetKind::Text, "c1")], + }; + // Applying the annotation facet keeps exactly the starred rows. + assert_eq!(matching_rows(&d, &cfg, &inputs).unwrap(), vec![0, 2]); + + // The text facet's counts reflect the annotation selection. + let rs = compute(&d, &cfg, &inputs, None).unwrap(); + assert_eq!(rs.matched_rows, 2); + let name = find(&rs, "name"); + assert_eq!(count(name, "Ada"), 1); + assert_eq!(count(name, "Cy"), 1); + assert!(name.buckets.iter().all(|b| b.key != "Bob")); // Bob (row1) filtered out + } + + #[test] + fn status_none_bucket_is_selectable() { + let (d, inputs) = annotated_doc(); + let mut ann = FacetSpec { + id: "ann".into(), + kind: FacetKind::Annotation, + column_id: None, + semantic: None, + selection: include(&[NONE_KEY]), + top_n: None, + search: None, + bins: None, + pinned: false, + collapsed: false, + width: None, + }; + ann.selection.mode = FacetMode::Include; + let cfg = FacetConfig { facets: vec![ann] }; + // Only row 4 has no annotation. + assert_eq!(matching_rows(&d, &cfg, &inputs).unwrap(), vec![4]); + } + + // ----- self-review regression tests ------------------------------------ + + fn ann_facet(id: &str) -> FacetSpec { + FacetSpec { + id: id.into(), + kind: FacetKind::Annotation, + column_id: None, + semantic: None, + selection: FacetSelection::default(), + top_n: None, + search: None, + bins: None, + pinned: false, + collapsed: false, + width: None, + } + } + + #[test] + fn truncated_scan_flags_status_facets_sampled() { + // A truncated scan (as an indexed document over FACET_SAMPLE_ROWS gets) + // under-counts status facets exactly like value facets, so their per-facet + // `sampled` flag — the one that drives the UI "≈" estimate mark — must be + // set, not just the top-level flag. `compute_scan` forces the sampled path + // without a 200k-row indexed fixture. + let (d, inputs) = annotated_doc(); // 5 rows + let cfg = FacetConfig { + facets: vec![ann_facet("ann"), spec("name", FacetKind::Text, "c1")], + }; + // Full scan: nothing sampled. + let full = compute_scan(&d, &cfg, &inputs, None, d.n_rows()).unwrap(); + assert!(!full.sampled); + assert!(!find(&full, "ann").sampled); + assert!(!find(&full, "name").sampled); + + // Truncated to the leading 3 rows: everything is an estimate. + let rs = compute_scan(&d, &cfg, &inputs, None, 3).unwrap(); + assert!(rs.sampled); + assert_eq!(rs.scanned_rows, 3); + assert!( + find(&rs, "ann").sampled, + "status facet must flag sampled when its counts are truncated" + ); + assert!(find(&rs, "name").sampled); + } + + #[test] + fn stale_diagnostics_issue_degrades_instead_of_failing() { + // A diagnostics report cached before a column delete can name a column + // that no longer exists ("whitespace:9"). `from_diagnostics` must skip it + // (like F42) rather than propagate `issue_rows`' out-of-range error and + // fail every other facet in the same compute call. + use crate::diagnostics::{DiagnosticIssue, DiagnosticsReport, Severity}; + let d = sample_doc(); // 3 columns + let stale = DiagnosticIssue { + id: "whitespace:9".into(), + kind: "whitespace".into(), + severity: Severity::Warning, + title: "Edge whitespace".into(), + description: String::new(), + affected_count: 0, + samples: Vec::new(), + suggested_action: None, + row_filterable: true, + }; + let report = DiagnosticsReport { + doc_id: 1, + revision: d.revision(), + source: Vec::new(), + current: vec![stale], + }; + let input = StatusInput::from_diagnostics(&d, &report).expect("stale issue must not error"); + // The dead category is dropped; the facet degrades to just its none bucket. + assert!(input.categories.is_empty()); + assert_eq!(input.none_label.as_deref(), Some("No issues")); + } + + #[test] + fn stale_crossval_rule_degrades_instead_of_failing() { + // A cached cross-column rule referencing a renamed/deleted column makes + // `violating_rows` error on name resolution; `from_validation` must swallow + // that (like F42) and still return a usable validation facet. + use crate::crossval::CrossRule; + let d = sample_doc(); + let rules = vec![CrossRule::ExactlyOne { + columns: vec!["ghost_a".into(), "ghost_b".into()], + }]; + let input = StatusInput::from_validation(&d, &rules, d.schema_issues()) + .expect("stale rule must not error"); + // Cross-val category present but degraded to empty; schema category always. + let crossval = input + .categories + .iter() + .find(|c| c.key == "crossval") + .expect("crossval category present"); + assert!(crossval.rows.is_empty()); + assert!(input.categories.iter().any(|c| c.key == "schema")); + } + + #[test] + fn selected_text_buckets_are_bounded() { + // A hand-edited / persisted saved view with a giant selection.values array + // must not turn into an unbounded bucket payload: the always-shown selected + // buckets are capped at MAX_TEXT_VALUES (plus at most top_n "rest" buckets). + let d = sample_doc(); + let mut s = spec("city", FacetKind::Text, "c0"); + let values: Vec = (0..MAX_TEXT_VALUES + 500) + .map(|i| format!("v{i}")) + .collect(); + s.selection = FacetSelection { + mode: FacetMode::Include, + values, + range: FacetRange::default(), + }; + s.top_n = Some(20); + let cfg = FacetConfig { facets: vec![s] }; + let rs = compute(&d, &cfg, &FacetInputs::default(), None).unwrap(); + let f = find(&rs, "city"); + assert!( + f.buckets.len() <= MAX_TEXT_VALUES + 20, + "selected buckets must stay bounded, got {}", + f.buckets.len() + ); + } +} diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index 9ba5afe..3c41243 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -48,6 +48,14 @@ mod error; pub mod excel; mod export; mod export_scope; +/// Multi-facet exploration (F39): the cross-filtering facet engine — the ten +/// facet types, single-pass per-facet count recomputation against the +/// other-facets population, the exact row-view producer that composes with the +/// filter/sort/export pipeline, the facets→filter-builder conversion and the +/// F12 named-view facet payload. Public like [`job`] so the command surface, +/// the F12 view persistence and the test harness treat it as a stable internal +/// API. +pub mod facets; mod filter; mod find; mod follow; @@ -424,6 +432,9 @@ pub fn run() { db_export::db_export_preview, db_export::start_db_export, db_export::db_export_report, + commands::compute_facets, + commands::apply_facets, + commands::convert_facets_to_filter, ]) .build(tauri::generate_context!()) .expect("error while running tauri application") diff --git a/src-tauri/src/project.rs b/src-tauri/src/project.rs index ba96f82..180c4f2 100644 --- a/src-tauri/src/project.rs +++ b/src-tauri/src/project.rs @@ -1577,6 +1577,7 @@ mod tests { column_widths: std::collections::HashMap::new(), wrap_text: false, highlight_rules: Vec::new(), + facets: None, } } diff --git a/src-tauri/src/settings.rs b/src-tauri/src/settings.rs index ffe5dac..17790a0 100644 --- a/src-tauri/src/settings.rs +++ b/src-tauri/src/settings.rs @@ -106,6 +106,12 @@ pub struct NamedView { /// decoration; carries no cell data). #[serde(default)] pub highlight_rules: Vec, + /// F39: multi-facet exploration configuration saved with this view — the + /// active facet panels, their selections and (non-computational) panel + /// layout. Faceting is non-destructive, so this carries no cell data; + /// restoring a view re-applies the facets exactly. `None` = no facets saved. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub facets: Option, } /// A reusable description of a recurring file format. @@ -574,6 +580,7 @@ mod tests { column_widths: std::collections::HashMap::from([("c1".to_string(), 240.0)]), wrap_text: true, highlight_rules: Vec::new(), + facets: None, }]; p.last_view_id = Some("v1".into()); settings.profiles.push(p); @@ -643,6 +650,7 @@ mod tests { column_widths: std::collections::HashMap::new(), wrap_text: false, highlight_rules: vec![rule], + facets: None, }]; settings.profiles.push(p); save_settings(dir.path(), &settings).unwrap(); @@ -657,6 +665,83 @@ mod tests { ); } + #[test] + fn view_facets_round_trip_and_default_none() { + use crate::facets::{ + FacetConfig, FacetKind, FacetMode, FacetRange, FacetSelection, FacetSpec, + }; + let dir = tempfile::tempdir().unwrap(); + let mut settings = AppSettings::default(); + let mut p = profile(); + + let mut view = NamedView { + id: "v1".into(), + name: "facet slice".into(), + filter: None, + filter_column_ids: Vec::new(), + sort_keys: Vec::new(), + hidden_column_ids: Vec::new(), + pinned_column_ids: Vec::new(), + column_order: Vec::new(), + column_widths: std::collections::HashMap::new(), + wrap_text: false, + highlight_rules: Vec::new(), + facets: None, + }; + // Defaults to None (a view without facets saved). + assert!(view.facets.is_none()); + + view.facets = Some(FacetConfig { + facets: vec![ + FacetSpec { + id: "city".into(), + kind: FacetKind::Text, + column_id: Some("c0".into()), + semantic: None, + selection: FacetSelection { + mode: FacetMode::Include, + values: vec!["NYC".into(), "LA".into()], + range: FacetRange::default(), + }, + top_n: Some(25), + search: None, + bins: None, + pinned: true, + collapsed: false, + width: Some(220.0), + }, + FacetSpec { + id: "age".into(), + kind: FacetKind::Number, + column_id: Some("c1".into()), + semantic: None, + selection: FacetSelection { + mode: FacetMode::Exclude, + values: Vec::new(), + range: FacetRange { + min: Some("18".into()), + max: Some("65".into()), + }, + }, + top_n: None, + search: None, + bins: Some(12), + pinned: false, + collapsed: true, + width: None, + }, + ], + }); + p.named_views = vec![view]; + settings.profiles.push(p); + save_settings(dir.path(), &settings).unwrap(); + let loaded = load_settings(dir.path()); + assert_eq!( + loaded.profiles[0].named_views[0].facets, + settings.profiles[0].named_views[0].facets + ); + } + #[test] fn semantic_overrides_round_trip_and_default_empty() { use crate::semantic::SemanticType; diff --git a/src/App.tsx b/src/App.tsx index d429e67..ad0187a 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -26,6 +26,7 @@ import { EmptyState } from "./components/EmptyState"; import { EncodingIssuesDialog } from "./components/EncodingIssuesDialog"; import { ExportDialog } from "./components/ExportDialog"; import { ExternalChangeDialog } from "./components/ExternalChangeDialog"; +import { FacetsPanel } from "./components/FacetsPanel"; import { FilterDialog } from "./components/FilterDialog"; import { FindBar } from "./components/FindBar"; import { FollowBar } from "./components/FollowBar"; @@ -315,6 +316,7 @@ export default function App() { {changesOpen && meta && } {annotationsPanelOpen && meta && } {meta && } + {meta && } {meta && } diff --git a/src/components/FacetsPanel.tsx b/src/components/FacetsPanel.tsx new file mode 100644 index 0000000..c8c1c27 --- /dev/null +++ b/src/components/FacetsPanel.tsx @@ -0,0 +1,551 @@ +import { useEffect, useRef, useState } from "react"; + +import { + FACET_KIND_LABELS, + SEMANTIC_FACET_TYPES, + displayOrder, + formatBucketCount, + formatPopulation, + isColumnScoped, + selectionActive, +} from "../lib/facets"; +import { useActiveMeta, useStore } from "../store/useStore"; +import type { FacetKind, FacetResult, FacetSpec, SemanticType } from "../types"; +import { BarChart, ChevronDown, ChevronUp, Close } from "./Icons"; + +/** Facet kinds offered in the add-facet picker, in menu order. */ +const COLUMN_FACET_KINDS: FacetKind[] = [ + "text", + "number", + "date", + "boolean", + "nullability", + "semantic", +]; +const STATUS_FACET_KINDS: FacetKind[] = ["diagnostics", "validation", "duplicate", "annotation"]; + +/** + * Multi-facet exploration panel (F39). Several facets are active at once; the + * grid row view is driven by the AND across panels and the OR (with include / + * exclude) inside one. Every facet's counts are recomputed against the + * population filtered by all the OTHER facets, so counts always reflect the + * current cross-filter. Faceting is non-destructive — it never dirties the + * document — and integrates with the existing row-view pipeline, so visible-row + * export just works. + */ +export function FacetsPanel() { + const meta = useActiveMeta(); + const facets = useStore((s) => s.facets); + const setOpen = useStore((s) => s.setFacetsOpen); + const addFacet = useStore((s) => s.addFacet); + const clearAll = useStore((s) => s.clearAllFacets); + const convertToFilter = useStore((s) => s.convertFacetsToFilter); + const syncFacets = useStore((s) => s.syncFacets); + const setModal = useStore((s) => s.setModal); + + // Recompute when the panel opens or the active document changes (selection + // changes drive their own debounced sync from the store). + useEffect(() => { + if (facets.open && meta) void syncFacets(); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [facets.open, meta?.id]); + + if (!meta || !facets.open) return null; + + const results = facets.results; + const anyActive = facets.config.facets.some((f) => selectionActive(f.selection)); + + return ( + + ); +} + +/** The add-facet control: a kind picker plus a column (or semantic type) picker. */ +function AddFacetBar({ + meta, + onAdd, +}: { + meta: NonNullable>; + onAdd: (kind: FacetKind, columnId?: string | null, semantic?: SemanticType | null) => void; +}) { + const [kind, setKind] = useState("text"); + const [column, setColumn] = useState(0); + const [semantic, setSemantic] = useState("email"); + const scoped = isColumnScoped(kind); + + const add = () => { + if (scoped) { + const columnId = meta.columnIds[column] ?? null; + onAdd(kind, columnId, kind === "semantic" ? semantic : null); + } else { + onAdd(kind); + } + }; + + return ( +
+ + + {scoped && ( + + )} + + {kind === "semantic" && ( + + )} + + +
+ ); +} + +/** One facet panel: header (title / mode / pin / collapse / copy / remove) plus + * a type-appropriate body. Draggable to reorder. */ +function FacetCard({ + spec, + result, + headers, + columnIds, +}: { + spec: FacetSpec; + result: FacetResult | null; + headers: string[]; + columnIds: string[]; +}) { + const config = useStore((s) => s.facets.config); + const removeFacet = useStore((s) => s.removeFacet); + const patchFacet = useStore((s) => s.patchFacet); + const toggleMode = useStore((s) => s.toggleFacetMode); + const clearFacet = useStore((s) => s.clearFacet); + const copyFacet = useStore((s) => s.copyFacet); + const reorderFacet = useStore((s) => s.reorderFacet); + const dragId = useRef(null); + + const title = isColumnScoped(spec.kind) + ? `${columnName(headers, columnIds, spec.columnId)} · ${FACET_KIND_LABELS[spec.kind]}` + : FACET_KIND_LABELS[spec.kind]; + const active = selectionActive(spec.selection); + + const onDrop = (targetId: string) => { + const from = config.facets.findIndex((f) => f.id === dragId.current); + const to = config.facets.findIndex((f) => f.id === targetId); + if (from >= 0 && to >= 0) reorderFacet(from, to); + dragId.current = null; + }; + + return ( +
  • (dragId.current = spec.id)} + onDragOver={(e) => e.preventDefault()} + onDrop={() => onDrop(spec.id)} + className={`rounded-lg border ${ + active + ? "border-violet-300 dark:border-violet-500/40" + : "border-zinc-200 dark:border-zinc-800" + } bg-white dark:bg-zinc-900/40`} + > +
    + + + {title} + + {result?.sampled && ( + + est + + )} + + + + +
    + + {!spec.collapsed && ( +
    + {result?.unresolved ? ( + + ) : ( + + )} + {active && ( + + )} +
    + )} +
    • + ); +} + +function FacetBody({ spec, result }: { spec: FacetSpec; result: FacetResult | null }) { + if (!result) return

    Computing…

    ; + switch (spec.kind) { + case "number": + case "date": + return ; + default: + // text / boolean / nullability / semantic / status all render value lists. + return ; + } +} + +function UnresolvedNote({ kind }: { kind: FacetKind }) { + const msg = + kind === "diagnostics" + ? "Run a diagnostics scan to enable this facet." + : kind === "duplicate" + ? "Run a duplicate scan (Duplicates dialog) to enable this facet." + : "This facet's column no longer exists in the current layout."; + return

    {msg}

    ; +} + +/** Text / boolean / nullability / semantic / status: a checkbox value list with + * cross-filtered counts, a proportion bar, and (for text) a search box. */ +function ValueListBody({ spec, result }: { spec: FacetSpec; result: FacetResult }) { + const toggleValue = useStore((s) => s.toggleFacetValue); + const patchFacet = useStore((s) => s.patchFacet); + const isText = spec.kind === "text"; + const maxCount = result.buckets.reduce((m, b) => Math.max(m, b.count), 0) || 1; + + return ( +
    + {isText && ( + patchFacet(spec.id, { search: e.target.value || null })} + placeholder={ + result.distinct != null + ? `Search ${result.distinct.toLocaleString()} values…` + : "Search…" + } + className="mb-1.5 w-full rounded border border-zinc-300 bg-transparent px-1.5 py-1 text-xs outline-none focus:border-violet-500 dark:border-zinc-700" + /> + )} + {result.buckets.length === 0 ? ( +

    No values.

    + ) : ( +
      + {result.buckets.map((b) => ( +
    • toggleValue(spec.id, b.key)} + > + + + + {b.label === "" ? (empty) : b.label} + + + {formatBucketCount(b.count, result.sampled)} + +
      • + ))} +
    + )} + {result.truncated && ( +

    + Showing top values only — this column has more distinct values than can be listed. +

    + )} +
    + ); +} + +// The backend labels a histogram bin "lo – hi" (spaced en dash); split on the +// spaced dash so date/number labels with internal hyphens stay intact. +const RANGE_SEP = /\s+[–-]\s+/; + +/** Number / date: a pure-CSS histogram (click a bar to select its range) plus + * min/max inputs for a precise continuous range. */ +function RangeBody({ spec, result }: { spec: FacetSpec; result: FacetResult }) { + const setRange = useStore((s) => s.setFacetRange); + const range = result.range; + const selMin = range?.selectedMin ?? ""; + const selMax = range?.selectedMax ?? ""; + const [min, setMin] = useState(selMin); + const [max, setMax] = useState(selMax); + + // Keep local inputs in sync when the selection changes elsewhere (bar click, + // clear, view restore). + useEffect(() => setMin(selMin), [selMin]); + useEffect(() => setMax(selMax), [selMax]); + + const maxCount = result.buckets.reduce((m, b) => Math.max(m, b.count), 0) || 1; + + const barSelected = (lo?: number, hi?: number) => { + if (!selectionActive(spec.selection) || lo == null || hi == null) return false; + return overlapsSelection(spec, lo, hi); + }; + + const clickBar = (label: string) => { + const parts = label.split(RANGE_SEP); + if (parts.length === 2) setRange(spec.id, parts[0].trim(), parts[1].trim()); + }; + + return ( +
    + {range && (range.min != null || range.max != null) && ( +

    + Range {range.min ?? "—"} to {range.max ?? "—"} +

    + )} + {result.buckets.length > 0 && ( +
    + {result.buckets.map((b) => ( +
    + )} +
    + setMin(e.target.value)} + onBlur={() => setRange(spec.id, min || null, max || null)} + onKeyDown={(e) => e.key === "Enter" && setRange(spec.id, min || null, max || null)} + placeholder={range?.min ?? "min"} + className={rangeInputCls} + /> + to + setMax(e.target.value)} + onBlur={() => setRange(spec.id, min || null, max || null)} + onKeyDown={(e) => e.key === "Enter" && setRange(spec.id, min || null, max || null)} + placeholder={range?.max ?? "max"} + className={rangeInputCls} + /> +
    +
    + ); +} + +/** Whether a histogram bar [lo,hi] overlaps the facet's selected numeric band. + * The selection bounds are strings in the column's format, so we compare on the + * bar's numeric edges only when both selection bounds parse as finite numbers; + * date columns fall back to no highlight (still fully usable via the inputs). */ +function overlapsSelection(spec: FacetSpec, lo: number, hi: number): boolean { + const min = spec.selection.range.min; + const max = spec.selection.range.max; + const lowerOk = min == null || !Number.isFinite(Number(min)) || hi >= Number(min); + const upperOk = max == null || !Number.isFinite(Number(max)) || lo <= Number(max); + const anyNumeric = + (min != null && Number.isFinite(Number(min))) || (max != null && Number.isFinite(Number(max))); + return anyNumeric && lowerOk && upperOk; +} + +/** Resolve a facet's stable column ID to its current header label (falls back + * to the ID when the column no longer exists — the card also flags unresolved). */ +function columnName(headers: string[], columnIds: string[], columnId?: string | null): string { + if (!columnId) return "—"; + const idx = columnIds.indexOf(columnId); + if (idx < 0) return columnId; + return headers[idx]?.trim() || `Column ${idx + 1}`; +} + +const selectCls = + "rounded border border-zinc-300 bg-transparent px-1.5 py-1 text-xs outline-none focus:border-violet-500 dark:border-zinc-700"; +const rangeInputCls = + "w-24 rounded border border-zinc-300 bg-transparent px-1.5 py-1 tabular-nums outline-none focus:border-violet-500 dark:border-zinc-700"; diff --git a/src/components/Toolbar.tsx b/src/components/Toolbar.tsx index 39ed2f3..a5b3666 100644 --- a/src/components/Toolbar.tsx +++ b/src/components/Toolbar.tsx @@ -74,6 +74,8 @@ export function Toolbar() { const setDiagnosticsOpen = useStore((s) => s.setDiagnosticsOpen); const explorerOpen = useStore((s) => s.explorer.open); const setExplorerOpen = useStore((s) => s.setExplorerOpen); + const facetsOpen = useStore((s) => s.facets.open); + const setFacetsOpen = useStore((s) => s.setFacetsOpen); const annotationsPanelOpen = useStore((s) => s.annotationsPanelOpen); const setAnnotationsPanelOpen = useStore((s) => s.setAnnotationsPanelOpen); @@ -178,6 +180,14 @@ export function Toolbar() { active: explorerOpen, disabled: !hasDoc, }, + { + label: "Facets", + title: "Multi-facet exploration (cross-filtered)", + icon: , + onClick: () => setFacetsOpen(!facetsOpen), + active: facetsOpen, + disabled: !hasDoc, + }, { label: "Diagnostics", title: "Data-fidelity diagnostics", diff --git a/src/lib/commandDefs.ts b/src/lib/commandDefs.ts index e752528..5a3677c 100644 --- a/src/lib/commandDefs.ts +++ b/src/lib/commandDefs.ts @@ -795,6 +795,30 @@ function staticCommands(): AppCommand[] { unavailableReason: needsDoc, run: () => state().setExplorerOpen(!state().explorer.open), }, + { + id: "view.facets", + title: "Toggle facets", + keywords: [ + "facet", + "faceted search", + "explore", + "cross-filter", + "dimensions", + "drill down", + "slice", + ], + category: "View", + unavailableReason: needsDoc, + run: () => state().setFacetsOpen(!state().facets.open), + }, + { + id: "view.facetsToFilter", + title: "Convert facets to filter", + keywords: ["facet", "filter", "convert", "build filter"], + category: "View", + unavailableReason: needsDoc, + run: () => void state().convertFacetsToFilter(), + }, { id: "view.recordForm", title: "Toggle record form", diff --git a/src/lib/facets.test.ts b/src/lib/facets.test.ts new file mode 100644 index 0000000..72a93b2 --- /dev/null +++ b/src/lib/facets.test.ts @@ -0,0 +1,186 @@ +import { describe, expect, it } from "vitest"; + +import type { FacetConversion, FacetResult, FacetSelection, FilterGroup } from "../types"; +import { + addFacet, + anyFacetActive, + clearSelection, + createFacetSpec, + countConditions, + describeDropped, + displayOrder, + facetClipboardText, + formatBucketCount, + formatCount, + formatPopulation, + moveFacet, + removeFacet, + selectionActive, + setMode, + setRange, + summarizeConversion, + toggleMode, + toggleValue, + updateSelection, +} from "./facets"; + +function sel(over: Partial = {}): FacetSelection { + return { mode: "include", values: [], range: {}, ...over }; +} + +describe("selection-state reducers", () => { + it("toggleValue adds then removes a value (immutably)", () => { + const a = sel(); + const b = toggleValue(a, "NYC"); + expect(b.values).toEqual(["NYC"]); + expect(a.values).toEqual([]); // original untouched + const c = toggleValue(b, "LA"); + expect(c.values).toEqual(["NYC", "LA"]); + const d = toggleValue(c, "NYC"); + expect(d.values).toEqual(["LA"]); + }); + + it("setMode / toggleMode flip include ⇄ exclude", () => { + expect(setMode(sel(), "exclude").mode).toBe("exclude"); + expect(toggleMode(sel({ mode: "include" })).mode).toBe("exclude"); + expect(toggleMode(sel({ mode: "exclude" })).mode).toBe("include"); + }); + + it("setRange trims bounds and clears blank ones", () => { + const r = setRange(sel(), " 25 ", ""); + expect(r.range).toEqual({ min: "25", max: null }); + const r2 = setRange(sel(), null, "100"); + expect(r2.range).toEqual({ min: null, max: "100" }); + }); + + it("clearSelection empties values+range but keeps the mode", () => { + const start = sel({ mode: "exclude", values: ["x"], range: { min: "1", max: "2" } }); + const cleared = clearSelection(start); + expect(cleared).toEqual({ mode: "exclude", values: [], range: {} }); + }); + + it("selectionActive reflects values or either range bound", () => { + expect(selectionActive(sel())).toBe(false); + expect(selectionActive(sel({ values: ["a"] }))).toBe(true); + expect(selectionActive(sel({ range: { min: "1", max: null } }))).toBe(true); + expect(selectionActive(sel({ range: { min: " ", max: null } }))).toBe(false); + }); +}); + +describe("config updaters", () => { + it("add / remove / updateSelection / anyFacetActive compose", () => { + let cfg = { facets: [] as ReturnType[] }; + const city = createFacetSpec("text", "c0"); + cfg = addFacet(cfg, city); + expect(cfg.facets).toHaveLength(1); + expect(anyFacetActive(cfg)).toBe(false); + + cfg = updateSelection(cfg, city.id, (s) => toggleValue(s, "NYC")); + expect(anyFacetActive(cfg)).toBe(true); + expect(cfg.facets[0].selection.values).toEqual(["NYC"]); + + cfg = removeFacet(cfg, city.id); + expect(cfg.facets).toHaveLength(0); + }); + + it("moveFacet reorders and is a no-op for out-of-range / equal indices", () => { + const a = createFacetSpec("text", "c0"); + const b = createFacetSpec("number", "c1"); + const c = createFacetSpec("boolean", "c2"); + const cfg = { facets: [a, b, c] }; + expect(moveFacet(cfg, 0, 2).facets.map((f) => f.id)).toEqual([b.id, c.id, a.id]); + expect(moveFacet(cfg, 2, 0).facets.map((f) => f.id)).toEqual([c.id, a.id, b.id]); + expect(moveFacet(cfg, 1, 1)).toBe(cfg); + expect(moveFacet(cfg, 5, 0)).toBe(cfg); + }); + + it("displayOrder floats pinned facets to the top preserving order", () => { + const a = { ...createFacetSpec("text", "c0"), pinned: false }; + const b = { ...createFacetSpec("number", "c1"), pinned: true }; + const c = { ...createFacetSpec("boolean", "c2"), pinned: false }; + const order = displayOrder({ facets: [a, b, c] }).map((f) => f.id); + expect(order).toEqual([b.id, a.id, c.id]); + }); +}); + +describe("count formatting", () => { + it("formatCount groups digits deterministically", () => { + expect(formatCount(0)).toBe("0"); + expect(formatCount(42)).toBe("42"); + expect(formatCount(1000)).toBe("1,000"); + expect(formatCount(1234567)).toBe("1,234,567"); + expect(formatCount(-2500)).toBe("-2,500"); + }); + + it("formatBucketCount marks sampled counts with ≈", () => { + expect(formatBucketCount(1200, false)).toBe("1,200"); + expect(formatBucketCount(1200, true)).toBe("≈ 1,200"); + }); + + it("formatPopulation marks an estimated total", () => { + expect(formatPopulation(3, 1000, false)).toBe("3 of 1,000 rows"); + expect(formatPopulation(3, 1000, true)).toBe("3 of ≈ 1,000 rows"); + }); +}); + +describe("clipboard", () => { + it("facetClipboardText emits value\\tcount lines", () => { + const result = { + buckets: [ + { key: "NYC", label: "NYC", count: 3, selected: true }, + { key: "LA", label: "LA", count: 2, selected: false }, + ], + } as FacetResult; + expect(facetClipboardText(result)).toBe("NYC\t3\nLA\t2"); + }); +}); + +describe("conversion mapping", () => { + function group(nodes: FilterGroup["nodes"]): FilterGroup { + return { type: "group", id: "g", conjunction: "and", nodes }; + } + + it("countConditions walks nested groups", () => { + const tree = group([ + { type: "condition", id: "1", column: 0, op: "equals", value: "NYC", caseSensitive: true }, + group([ + { type: "condition", id: "2", column: 1, op: "gte", value: "25", caseSensitive: false }, + { type: "condition", id: "3", column: 1, op: "lte", value: "40", caseSensitive: false }, + ]), + ]); + expect(countConditions(tree)).toBe(3); + }); + + it("summarizeConversion reports empty when no conditions, and carries dropped", () => { + const empty: FacetConversion = { + filter: group([]), + dropped: [{ id: "sem", reason: "semantic facets have no column-filter equivalent" }], + }; + const s = summarizeConversion(empty); + expect(s.conditionCount).toBe(0); + expect(s.empty).toBe(true); + expect(s.dropped).toHaveLength(1); + + const nonEmpty: FacetConversion = { + filter: group([ + { type: "condition", id: "1", column: 0, op: "equals", value: "x", caseSensitive: true }, + ]), + dropped: [], + }; + const s2 = summarizeConversion(nonEmpty); + expect(s2.empty).toBe(false); + expect(s2.conditionCount).toBe(1); + }); + + it("describeDropped pluralizes (or is empty)", () => { + expect(describeDropped([])).toBe(""); + expect(describeDropped([{ id: "a", reason: "" }])).toContain("1 facet"); + expect(describeDropped([{ id: "a", reason: "" }])).toContain("was"); + expect( + describeDropped([ + { id: "a", reason: "" }, + { id: "b", reason: "" }, + ]), + ).toContain("2 facets"); + }); +}); diff --git a/src/lib/facets.ts b/src/lib/facets.ts new file mode 100644 index 0000000..653e21e --- /dev/null +++ b/src/lib/facets.ts @@ -0,0 +1,288 @@ +// Pure helpers for multi-facet exploration (F39): selection-state reducers, +// count formatting, the facets → filter conversion summary, and small config +// updaters. No React, no store, no invoke — everything here is unit-tested. + +import type { + DroppedFacet, + FacetConfig, + FacetConversion, + FacetKind, + FacetMode, + FacetResult, + FacetSelection, + FacetSpec, + FilterGroup, + FilterNode, + SemanticType, +} from "../types"; + +// ----- facet metadata ------------------------------------------------------- + +/** Human labels for the picker and card headers. */ +export const FACET_KIND_LABELS: Record = { + text: "Values", + number: "Number range", + date: "Date range", + boolean: "True / false", + nullability: "Blank / null / invalid", + semantic: "Semantic type", + diagnostics: "Diagnostics status", + validation: "Validation status", + duplicate: "Duplicate status", + annotation: "Bookmarks / flags / tags", +}; + +/** Column-scoped facets need a column; the four status facets do not. */ +export const COLUMN_SCOPED_KINDS: ReadonlySet = new Set([ + "text", + "number", + "date", + "boolean", + "nullability", + "semantic", +]); + +/** The row-level status facets, sourced from the analysis caches. */ +export const STATUS_KINDS: ReadonlySet = new Set([ + "diagnostics", + "validation", + "duplicate", + "annotation", +]); + +/** Facet kinds that convert cleanly to a filter-builder condition. Boolean, + * semantic and the status facets have no faithful column-filter equivalent, and + * nullability only converts for a single blank/value include. */ +export const CONVERTIBLE_KINDS: ReadonlySet = new Set([ + "text", + "number", + "date", + "nullability", +]); + +/** Semantic types offered when adding a semantic facet (those with a matcher). */ +export const SEMANTIC_FACET_TYPES: SemanticType[] = [ + "email", + "url", + "uuid", + "ipv4", + "ipv6", + "json", + "percentage", + "currency", + "phoneNumber", + "postalCode", +]; + +export function isColumnScoped(kind: FacetKind): boolean { + return COLUMN_SCOPED_KINDS.has(kind); +} + +// ----- spec construction ---------------------------------------------------- + +let facetSeq = 0; + +/** Unique, persistence-safe facet panel id. */ +export function newFacetId(): string { + facetSeq += 1; + return `facet-${Date.now().toString(36)}-${facetSeq}${Math.random().toString(36).slice(2, 5)}`; +} + +/** A fresh, empty selection (include mode, no values, no range). */ +export function emptySelection(): FacetSelection { + return { mode: "include", values: [], range: {} }; +} + +/** Build a default facet spec for a kind (+ column / semantic type as needed). */ +export function createFacetSpec( + kind: FacetKind, + columnId?: string | null, + semantic?: SemanticType | null, +): FacetSpec { + return { + id: newFacetId(), + kind, + columnId: columnId ?? null, + semantic: semantic ?? null, + selection: emptySelection(), + pinned: false, + collapsed: false, + }; +} + +// ----- selection-state reducers (pure) -------------------------------------- + +function hasBound(value: string | null | undefined): boolean { + return typeof value === "string" && value.trim() !== ""; +} + +/** Whether a selection is currently narrowing the population. */ +export function selectionActive(sel: FacetSelection): boolean { + return sel.values.length > 0 || hasBound(sel.range.min) || hasBound(sel.range.max); +} + +/** Whether ANY facet in the config carries an active selection. */ +export function anyFacetActive(config: FacetConfig): boolean { + return config.facets.some((f) => selectionActive(f.selection)); +} + +/** Toggle one categorical value in the OR set (add if absent, remove if present). */ +export function toggleValue(sel: FacetSelection, key: string): FacetSelection { + const has = sel.values.includes(key); + return { + ...sel, + values: has ? sel.values.filter((v) => v !== key) : [...sel.values, key], + }; +} + +/** Replace the whole value set (e.g. "select all" / "clear values"). */ +export function setValues(sel: FacetSelection, values: string[]): FacetSelection { + return { ...sel, values: [...values] }; +} + +/** Set the include/exclude mode. */ +export function setMode(sel: FacetSelection, mode: FacetMode): FacetSelection { + return { ...sel, mode }; +} + +/** Flip include ⇄ exclude. */ +export function toggleMode(sel: FacetSelection): FacetSelection { + return { ...sel, mode: sel.mode === "include" ? "exclude" : "include" }; +} + +/** Set the continuous range bounds (blank strings clear the bound). */ +export function setRange( + sel: FacetSelection, + min: string | null, + max: string | null, +): FacetSelection { + return { + ...sel, + range: { + min: hasBound(min) ? min!.trim() : null, + max: hasBound(max) ? max!.trim() : null, + }, + }; +} + +/** Clear the selection but keep the mode (so a cleared exclude stays exclude). */ +export function clearSelection(sel: FacetSelection): FacetSelection { + return { mode: sel.mode, values: [], range: {} }; +} + +// ----- config updaters (pure) ----------------------------------------------- + +export function addFacet(config: FacetConfig, spec: FacetSpec): FacetConfig { + return { facets: [...config.facets, spec] }; +} + +export function removeFacet(config: FacetConfig, id: string): FacetConfig { + return { facets: config.facets.filter((f) => f.id !== id) }; +} + +export function updateFacet( + config: FacetConfig, + id: string, + fn: (spec: FacetSpec) => FacetSpec, +): FacetConfig { + return { facets: config.facets.map((f) => (f.id === id ? fn(f) : f)) }; +} + +/** Update just one facet's selection through a reducer. */ +export function updateSelection( + config: FacetConfig, + id: string, + fn: (sel: FacetSelection) => FacetSelection, +): FacetConfig { + return updateFacet(config, id, (f) => ({ ...f, selection: fn(f.selection) })); +} + +/** Move a facet from one index to another (drag reorder). */ +export function moveFacet(config: FacetConfig, from: number, to: number): FacetConfig { + const n = config.facets.length; + if (from < 0 || from >= n || to < 0 || to >= n || from === to) return config; + const facets = [...config.facets]; + const [moved] = facets.splice(from, 1); + facets.splice(to, 0, moved); + return { facets }; +} + +/** + * Display order for the panel: pinned facets first, then unpinned, each in the + * config's own (drag) order. The stored config order is left untouched so pin + * and reorder stay independent. + */ +export function displayOrder(config: FacetConfig): FacetSpec[] { + const pinned = config.facets.filter((f) => f.pinned); + const rest = config.facets.filter((f) => !f.pinned); + return [...pinned, ...rest]; +} + +// ----- count formatting ----------------------------------------------------- + +/** Group digits with commas, deterministically (locale-independent). */ +export function formatCount(n: number): string { + const neg = n < 0; + const digits = Math.abs(Math.trunc(n)).toString(); + const grouped = digits.replace(/\B(?=(\d{3})+(?!\d))/g, ","); + return neg ? `-${grouped}` : grouped; +} + +/** The "estimated" prefix for a sampled count (almost-equal sign + ASCII space, + * defined once so the UI and tests share one source of truth). */ +export const ESTIMATE_MARK = "≈ "; + +/** A bucket count, prefixed with the estimate mark when sampled. */ +export function formatBucketCount(count: number, sampled: boolean): string { + return `${sampled ? "≈ " : ""}${formatCount(count)}`; +} + +/** "3 of 1,000 rows" style population summary. */ +export function formatPopulation(matched: number, total: number, sampled: boolean): string { + const est = sampled ? "≈ " : ""; + return `${formatCount(matched)} of ${est}${formatCount(total)} rows`; +} + +// ----- clipboard ------------------------------------------------------------ + +/** Tab-separated "value\tcount" lines for the whole facet (copy values+counts). */ +export function facetClipboardText(result: FacetResult): string { + return result.buckets.map((b) => `${b.label}\t${b.count}`).join("\n"); +} + +// ----- conversion mapping --------------------------------------------------- + +/** Count the leaf conditions in a filter tree (recursively). */ +export function countConditions(group: FilterGroup): number { + let total = 0; + const walk = (nodes: FilterNode[]) => { + for (const node of nodes) { + if (node.type === "condition") total += 1; + else walk(node.nodes); + } + }; + walk(group.nodes); + return total; +} + +/** Summary of a facets → filter conversion for the UI: how many conditions it + * produced, whether it is empty (nothing convertible), and what was dropped. */ +export interface ConversionSummary { + conditionCount: number; + empty: boolean; + dropped: DroppedFacet[]; +} + +export function summarizeConversion(conv: FacetConversion): ConversionSummary { + const conditionCount = countConditions(conv.filter); + return { conditionCount, empty: conditionCount === 0, dropped: conv.dropped }; +} + +/** One-line human note about what a conversion dropped (empty string if none). */ +export function describeDropped(dropped: DroppedFacet[]): string { + if (dropped.length === 0) return ""; + const n = dropped.length; + return `${n} facet${n === 1 ? "" : "s"} had no filter equivalent and ${ + n === 1 ? "was" : "were" + } left out.`; +} diff --git a/src/lib/tauri.ts b/src/lib/tauri.ts index 9866d25..591b3ee 100644 --- a/src/lib/tauri.ts +++ b/src/lib/tauri.ts @@ -139,6 +139,9 @@ import type { DbExportSpec, DbExportPreview, DbExportResult, + FacetConfig, + FacetResultSet, + FacetConversion, } from "../types"; export const openFile = (path: string, options?: OpenOptions) => @@ -947,6 +950,50 @@ export const setFilter = (docId: number, spec: FilterGroup) => export const clearFilter = (docId: number) => invoke("clear_filter", { docId }); +// ----- multi-facet exploration (F39) ---------------------------------------- + +/** + * Compute every facet's cross-filtered bucket counts against the current + * document (F39). Read-only; safe to recompute on every selection change. + * `dedup`/`dedupScope` resolve a duplicate-status facet (F05); omit them when + * no duplicate facet is present (or none has been configured yet). + */ +export const computeFacets = ( + docId: number, + config: FacetConfig, + expectedRevision: number, + dedup?: DedupSpec | null, + dedupScope?: ExportScope | null, +) => + invoke("compute_facets", { + docId, + config, + expectedRevision, + dedup: dedup ?? null, + dedupScope: dedupScope ?? null, + }); + +/** Drive the grid row view from the current facet selection (F39). A view + * operation — never dirties the document. Returns the refreshed meta. */ +export const applyFacets = ( + docId: number, + config: FacetConfig, + expectedRevision: number, + dedup?: DedupSpec | null, + dedupScope?: ExportScope | null, +) => + invoke("apply_facets", { + docId, + config, + expectedRevision, + dedup: dedup ?? null, + dedupScope: dedupScope ?? null, + }); + +/** Convert the active facets to the filter-builder tree (F39, one-way/lossy). */ +export const convertFacetsToFilter = (docId: number, config: FacetConfig) => + invoke("convert_facets_to_filter", { docId, config }); + /** F12: set (or clear, with empty keys) the non-destructive view sort. */ export const setViewSort = (docId: number, keys: SortKey[]) => invoke("set_view_sort", { docId, keys }); diff --git a/src/lib/views.ts b/src/lib/views.ts index f0324d2..61ce8b6 100644 --- a/src/lib/views.ts +++ b/src/lib/views.ts @@ -4,6 +4,7 @@ import type { DocumentMeta, + FacetConfig, FilterGroup, FilterNode, HighlightRule, @@ -87,11 +88,18 @@ export interface ViewSnapshotInput { wrapText: boolean; /** F42: the active document's conditional-highlighting rules, if any. */ highlightRules?: HighlightRule[]; + /** F39: the active document's multi-facet configuration, if any. */ + facets?: FacetConfig | null; } /** Build a persistable NamedView from the current state. */ export function snapshotView(input: ViewSnapshotInput): NamedView { const ids = input.meta.columnIds; + // Only persist a facet config when it actually carries panels. + const facets = + input.facets && input.facets.facets.length > 0 + ? { facets: input.facets.facets.map((f) => ({ ...f })) } + : null; return { id: input.id ?? newViewId(), name: input.name, @@ -104,6 +112,7 @@ export function snapshotView(input: ViewSnapshotInput): NamedView { columnWidths: widthsToIds(input.columnWidths, ids), wrapText: input.wrapText, highlightRules: (input.highlightRules ?? []).map((r) => ({ ...r })), + facets, }; } @@ -123,5 +132,8 @@ export function describeView(view: NamedView): string { `${view.highlightRules.length} highlight${view.highlightRules.length === 1 ? "" : "s"}`, ); } + if (view.facets && view.facets.facets.length > 0) { + parts.push(`${view.facets.facets.length} facet${view.facets.facets.length === 1 ? "" : "s"}`); + } return parts.length > 0 ? parts.join(" · ") : "layout only"; } diff --git a/src/store/useStore.test.ts b/src/store/useStore.test.ts index ffce684..7671163 100644 --- a/src/store/useStore.test.ts +++ b/src/store/useStore.test.ts @@ -5,6 +5,8 @@ import type { DbExportPreview, DictionaryView, DocumentMeta, + FacetConfig, + FacetResultSet, PlanEntry, ProjectMeta, ProjectOpenPlan, @@ -39,6 +41,9 @@ vi.mock("../lib/tauri", () => ({ annotationsView: vi.fn(), annotationsGetExport: vi.fn(), annotationsLoadExport: vi.fn().mockResolvedValue(undefined), + // F39 facets: apply drives the row view, compute returns cross-filtered counts. + applyFacets: vi.fn(), + computeFacets: vi.fn(), // F35 database export preview. dbExportPreview: vi.fn(), projectOpenApply: vi.fn(), @@ -691,6 +696,119 @@ describe("annotation persistence: sidecar vs project (F40)", () => { }); }); +describe("facet sync stale-response guard (F39)", () => { + interface Deferred { + promise: Promise; + resolve: (value: T) => void; + } + function deferred(): Deferred { + let resolve!: (value: T) => void; + const promise = new Promise((r) => { + resolve = r; + }); + return { promise, resolve }; + } + // Flush microtasks so a syncFacets awaiting a just-resolved deferred can run + // its next step before we resolve the following one. + const flush = () => new Promise((r) => setTimeout(r, 0)); + + function textConfig(id: string, value: string): FacetConfig { + return { + facets: [ + { + id, + kind: "text", + columnId: "c0", + selection: { mode: "include", values: [value], range: {} }, + pinned: false, + collapsed: false, + }, + ], + }; + } + + function resultSet(matchedRows: number): FacetResultSet { + return { + revision: 2, + matchedRows, + totalRows: 100, + scannedRows: 100, + sampled: false, + facets: [], + }; + } + + const initialFacetsState = { + open: true, + config: { facets: [] } as FacetConfig, + results: null, + loading: false, + error: null, + applied: false, + dedupContext: null, + }; + + beforeEach(() => { + vi.mocked(api.applyFacets).mockReset(); + vi.mocked(api.computeFacets).mockReset(); + useStore.setState({ + tabs: [meta(1)], + activeId: 1, + facets: { ...initialFacetsState }, + }); + }); + + it("drops a slower response for an older selection instead of clobbering the newer one", async () => { + const applyDeferreds: Deferred[] = []; + const computeDeferreds: Deferred[] = []; + vi.mocked(api.applyFacets).mockImplementation(() => { + const d = deferred(); + applyDeferreds.push(d); + return d.promise; + }); + vi.mocked(api.computeFacets).mockImplementation(() => { + const d = deferred(); + computeDeferreds.push(d); + return d.promise; + }); + + // Sync A (selection "NYC") starts and blocks in applyFacets. + useStore.setState({ + facets: { ...useStore.getState().facets, config: textConfig("f", "NYC") }, + }); + const syncA = useStore.getState().syncFacets(); + + // The user changes the selection to "LA"; sync B starts and blocks too. + useStore.setState({ facets: { ...useStore.getState().facets, config: textConfig("f", "LA") } }); + const syncB = useStore.getState().syncFacets(); + + expect(applyDeferreds).toHaveLength(2); + + // Newer sync B resolves fully first (fast backend for the newer config). + const metaB: DocumentMeta = { ...meta(1), revision: 2 }; + applyDeferreds[1].resolve(metaB); + await flush(); + const resultsB = resultSet(11); + computeDeferreds[0].resolve(resultsB); + await syncB; + + // State now reflects B. + expect(useStore.getState().facets.results).toEqual(resultsB); + expect(useStore.getState().facets.applied).toBe(true); + expect(useStore.getState().tabs[0].revision).toBe(2); + + // Now the STALE sync A finally returns from applyFacets. It must bail: no + // reloadDoc (revision stays B's 2, not A's 99) and no compute call for A. + const metaA: DocumentMeta = { ...meta(1), revision: 99 }; + applyDeferreds[0].resolve(metaA); + await syncA; + + expect(useStore.getState().tabs[0].revision).toBe(2); // A did not clobber + expect(useStore.getState().facets.results).toEqual(resultsB); // still B + expect(api.computeFacets).toHaveBeenCalledTimes(1); // A bailed before compute + }); +}); + describe("database export preview invalidation (F35)", () => { const exportForm = (over: Partial = {}): ExportForm => ({ path: "/tmp/out.sqlite", diff --git a/src/store/useStore.ts b/src/store/useStore.ts index ae6abf0..d028c13 100644 --- a/src/store/useStore.ts +++ b/src/store/useStore.ts @@ -1,5 +1,6 @@ import { create } from "zustand"; import { ask, open as openFileDialog, save as saveFileDialog } from "@tauri-apps/plugin-dialog"; +import { writeText as writeClipboard } from "@tauri-apps/plugin-clipboard-manager"; import { getCurrentWindow } from "@tauri-apps/api/window"; import * as api from "../lib/tauri"; @@ -18,6 +19,23 @@ import { type ColumnLayout, } from "../lib/viewProjection"; import { hydrateFilter, snapshotView, uniqueViewName, upsertView } from "../lib/views"; +import { + addFacet as addFacetToConfig, + anyFacetActive, + clearSelection, + createFacetSpec, + describeDropped, + facetClipboardText, + moveFacet, + removeFacet as removeFacetFromConfig, + setMode, + setRange, + summarizeConversion, + toggleMode, + toggleValue, + updateFacet, + updateSelection, +} from "../lib/facets"; import { annotationExportName } from "../lib/annotations"; import { currentOpenOptions, fingerprintKey } from "../lib/reopen"; import { defaultImportOptions } from "../lib/jsonImport"; @@ -146,6 +164,11 @@ import type { DbRefreshStatus, DbExportPreview, DbExportResult, + FacetConfig, + FacetResultSet, + FacetSpec, + FacetKind, + FacetMode, } from "../types"; import type { GatingWarning } from "../lib/project"; import { clampRecord, type RecordDraft } from "../lib/recordForm"; @@ -160,6 +183,14 @@ const MAX_RECENT = 10; let statsTimer: ReturnType | null = null; // Debounce timer for the (backend-computed) per-column summaries. let summariesTimer: ReturnType | null = null; +// Debounce timer for the (backend-computed) facet counts + row view (F39), so a +// burst of value toggles coalesces into one apply+compute round-trip. +let facetsTimer: ReturnType | null = null; +// Monotonic request token for facet syncs (F39). Each `syncFacets` captures the +// value it bumped this to; after every await it re-checks that it is still the +// newest in-flight sync and bails otherwise, so a slower response for an older +// selection can never call `reloadDoc`/write results on top of a newer one. +let facetSyncSeq = 0; /** Find-match cap for indexed read-only documents (F10). */ export const INDEXED_FIND_LIMIT = 5000; @@ -882,6 +913,37 @@ const initialExplorer: ExplorerState = { error: null, }; +/** + * Multi-facet exploration panel state (F39), for the ACTIVE document. `config` + * is the set of facet panels and their selections; `results` are the last + * computed cross-filtered counts. Faceting is non-destructive — nothing here + * ever dirties the document — but an active selection DOES drive the grid row + * view (`applied`), exactly like a filter. Config is per-document (columns are + * referenced by stable ID), so it resets on tab switch while the panel stays + * open. `dedupContext` remembers the last duplicate-scan spec so a duplicate + * status facet can resolve without re-prompting. + */ +export interface FacetsUiState { + open: boolean; + config: FacetConfig; + results: FacetResultSet | null; + loading: boolean; + error: string | null; + /** Whether the current selection is driving the grid row view. */ + applied: boolean; + dedupContext: { spec: DedupSpec; scope: ExportScope } | null; +} + +const initialFacets: FacetsUiState = { + open: false, + config: { facets: [] }, + results: null, + loading: false, + error: null, + applied: false, + dedupContext: null, +}; + const initialFilter: FilterState = { open: false, spec: { @@ -1000,6 +1062,8 @@ interface Store { profileValidation: ProfileValidation | null; /** Column-explorer panel state (F05). */ explorer: ExplorerState; + /** Multi-facet exploration panel state (F39). */ + facets: FacetsUiState; /** Duplicate-finder state (F07). */ dedup: DedupState; /** Compare state (F09). */ @@ -1644,6 +1708,31 @@ interface Store { applyValueFilter: (value: string, mode: "only" | "exclude" | "and") => Promise; applyRangeFilter: (min: string | null, max: string | null) => Promise; + // multi-facet exploration (F39) + setFacetsOpen: (open: boolean) => void; + /** Replace the whole facet configuration (e.g. restoring a saved view). */ + setFacetConfig: (config: FacetConfig) => void; + /** Add a facet panel for a column (or a status facet). */ + addFacet: (kind: FacetKind, columnId?: string | null, semantic?: SemanticType | null) => void; + removeFacet: (id: string) => void; + /** Mutate one facet's spec (pin / collapse / width / search / top-n / bins). */ + patchFacet: (id: string, patch: Partial) => void; + /** Toggle one categorical value inside a facet's OR set. */ + toggleFacetValue: (id: string, key: string) => void; + setFacetMode: (id: string, mode: FacetMode) => void; + toggleFacetMode: (id: string) => void; + setFacetRange: (id: string, min: string | null, max: string | null) => void; + clearFacet: (id: string) => void; + /** Clear every facet's selection (keeps the panels). */ + clearAllFacets: () => void; + reorderFacet: (from: number, to: number) => void; + /** Recompute counts + drive the grid row view from the current selection. */ + syncFacets: () => Promise; + /** Copy a facet's values+counts to the clipboard (TSV). */ + copyFacet: (id: string) => Promise; + /** Convert the active facets to a filter and open the FilterDialog on it. */ + convertFacetsToFilter: () => Promise; + // file profiles (F08) saveProfiles: (profiles: FileProfile[]) => Promise; /** Apply a profile via the previewed reopen flow (safe for dirty docs). */ @@ -1813,6 +1902,9 @@ export const useStore = create((set, get) => { total: null, error: null, }, + // F39: the facet config is per-document (columns by stable ID); keep the + // panel open across switches but drop the previous doc's facets/results. + facets: { ...initialFacets, open: s.facets.open }, dedup: initialDedup, // Cluster reports are per-document; never let one leak across tabs. cluster: initialCluster, @@ -2020,6 +2112,14 @@ export const useStore = create((set, get) => { set({ error: String(e) }); } + // F39: restore the view's facet configuration (or clear it when the view + // has none). Faceting is view-only: it never dirties the document. The + // config resolves against the CURRENT columns by stable ID; a facet whose + // column no longer exists renders unresolved (never silently applied). + const restoredFacets: FacetConfig = view.facets + ? { facets: view.facets.facets.map((f) => ({ ...f })) } + : { facets: [] }; + set((s) => ({ columnLayout: layoutIsTrivial(layout) ? null : layout, wrapText: view.wrapText, @@ -2031,12 +2131,18 @@ export const useStore = create((set, get) => { viewSortKeys: appliedSort, highlight: { ...initialHighlight, rules: highlightRules, loaded: true }, highlightVersion: s.highlightVersion + 1, + facets: { ...s.facets, config: restoredFacets }, viewWarning: missing.length > 0 ? `This view references ${missing.length} column${missing.length === 1 ? "" : "s"} that no longer exist (deleted or from an older file layout). The rest of the view was applied; the view itself is unchanged.` : null, })); void get().loadHighlightCounts(); + // Apply the restored facets' row view + counts (only when the panel is open, + // or when a selection is active so the grid reflects the saved view). + if (restoredFacets.facets.length > 0 || get().facets.applied) { + void get().syncFacets(); + } if (persistLast && meta.path) { await persistViews(meta, (views) => views, view.id).catch(() => undefined); @@ -2084,6 +2190,21 @@ export const useStore = create((set, get) => { } }; + /** + * Replace the facet config (F39) and schedule a debounced recompute that both + * refreshes the cross-filtered counts and drives the grid row view. Setting + * the config is synchronous (the cards react immediately); the backend round + * trip is coalesced so rapid toggling stays responsive. + */ + const applyFacetConfig = (config: FacetConfig) => { + set((s) => ({ facets: { ...s.facets, config } })); + if (facetsTimer !== null) clearTimeout(facetsTimer); + facetsTimer = setTimeout(() => { + facetsTimer = null; + void get().syncFacets(); + }, 120); + }; + /** Run a structural mutation against the active doc with error handling. */ const mutate = async (fn: (id: number) => Promise, reload = true) => { const id = get().activeId; @@ -2581,6 +2702,7 @@ export const useStore = create((set, get) => { profileSuggestion: null, profileValidation: null, explorer: initialExplorer, + facets: initialFacets, dedup: initialDedup, compare: initialCompare, @@ -2794,6 +2916,7 @@ export const useStore = create((set, get) => { columnWidths: s.columnWidths, wrapText: s.wrapText, highlightRules: s.highlight.rules, + facets: s.facets.config, }); try { await persistViews(meta, (views) => upsertView(views, view), view.id); @@ -2821,6 +2944,7 @@ export const useStore = create((set, get) => { columnWidths: s.columnWidths, wrapText: s.wrapText, highlightRules: s.highlight.rules, + facets: s.facets.config, }); try { await persistViews(meta, (views) => upsertView(views, view), viewId); @@ -4153,6 +4277,7 @@ export const useStore = create((set, get) => { diagnosticsOpen: open ? false : s.diagnosticsOpen, changesOpen: open ? false : s.changesOpen, explorer: open ? { ...s.explorer, open: false } : s.explorer, + facets: open ? { ...s.facets, open: false } : s.facets, // One side panel at a time — the record form (F41) shares the rail. recordFormOpen: open ? false : s.recordFormOpen, })); @@ -4679,6 +4804,7 @@ export const useStore = create((set, get) => { // The side area shows one panel at a time. changesOpen: open ? false : s.changesOpen, explorer: open ? { ...s.explorer, open: false } : s.explorer, + facets: open ? { ...s.facets, open: false } : s.facets, annotationsPanelOpen: open ? false : s.annotationsPanelOpen, recordFormOpen: open ? false : s.recordFormOpen, })), @@ -4688,6 +4814,7 @@ export const useStore = create((set, get) => { changesOpen: open, diagnosticsOpen: open ? false : s.diagnosticsOpen, explorer: open ? { ...s.explorer, open: false } : s.explorer, + facets: open ? { ...s.facets, open: false } : s.facets, annotationsPanelOpen: open ? false : s.annotationsPanelOpen, recordFormOpen: open ? false : s.recordFormOpen, })), @@ -4714,6 +4841,7 @@ export const useStore = create((set, get) => { changesOpen: open ? false : s.changesOpen, diagnosticsOpen: open ? false : s.diagnosticsOpen, explorer: open ? { ...s.explorer, open: false } : s.explorer, + facets: open ? { ...s.facets, open: false } : s.facets, annotationsPanelOpen: open ? false : s.annotationsPanelOpen, }; }), @@ -6794,6 +6922,9 @@ export const useStore = create((set, get) => { const jobId = await api.startDuplicateScan(meta.id, spec, scope, meta.revision); set((s) => ({ dedup: { ...s.dedup, scanJobId: jobId, processed: 0, total: null, error: null }, + // F39: remember the spec/scope so a duplicate-status facet resolves + // against the same grouping without re-prompting. + facets: { ...s.facets, dedupContext: { spec, scope } }, })); consumeEarlyFinish(jobId); } catch (e) { @@ -6887,6 +7018,7 @@ export const useStore = create((set, get) => { // The side area shows one panel at a time. diagnosticsOpen: open ? false : s.diagnosticsOpen, recordFormOpen: open ? false : s.recordFormOpen, + facets: open ? { ...s.facets, open: false } : s.facets, })); if (open) void get().refreshExplorerProfile(); }, @@ -6952,6 +7084,151 @@ export const useStore = create((set, get) => { void get().refreshExplorerProfile(); }, + // ----- multi-facet exploration (F39) --------------------------------------- + + setFacetsOpen: (open) => { + set((s) => ({ + facets: { ...s.facets, open }, + // The side rail shows one panel at a time. + explorer: open ? { ...s.explorer, open: false } : s.explorer, + diagnosticsOpen: open ? false : s.diagnosticsOpen, + changesOpen: open ? false : s.changesOpen, + annotationsPanelOpen: open ? false : s.annotationsPanelOpen, + recordFormOpen: open ? false : s.recordFormOpen, + })); + if (open) void get().syncFacets(); + }, + + setFacetConfig: (config) => applyFacetConfig(config), + + addFacet: (kind, columnId, semantic) => + applyFacetConfig( + addFacetToConfig(get().facets.config, createFacetSpec(kind, columnId, semantic)), + ), + + removeFacet: (id) => applyFacetConfig(removeFacetFromConfig(get().facets.config, id)), + + patchFacet: (id, patch) => { + // Layout-only tweaks (pin/collapse/width) never need a backend round trip; + // display tuning (search/topN/bins) does, since it changes the buckets. + const next = updateFacet(get().facets.config, id, (f) => ({ ...f, ...patch })); + const recompute = + "search" in patch || "topN" in patch || "bins" in patch || "semantic" in patch; + if (recompute) applyFacetConfig(next); + else set((s) => ({ facets: { ...s.facets, config: next } })); + }, + + toggleFacetValue: (id, key) => + applyFacetConfig(updateSelection(get().facets.config, id, (sel) => toggleValue(sel, key))), + + setFacetMode: (id, mode) => + applyFacetConfig(updateSelection(get().facets.config, id, (sel) => setMode(sel, mode))), + + toggleFacetMode: (id) => + applyFacetConfig(updateSelection(get().facets.config, id, (sel) => toggleMode(sel))), + + setFacetRange: (id, min, max) => + applyFacetConfig(updateSelection(get().facets.config, id, (sel) => setRange(sel, min, max))), + + clearFacet: (id) => + applyFacetConfig(updateSelection(get().facets.config, id, (sel) => clearSelection(sel))), + + clearAllFacets: () => + applyFacetConfig({ + facets: get().facets.config.facets.map((f) => ({ + ...f, + selection: clearSelection(f.selection), + })), + }), + + reorderFacet: (from, to) => applyFacetConfig(moveFacet(get().facets.config, from, to)), + + syncFacets: async () => { + const meta = activeMeta(); + const { facets } = get(); + // Not gated on the panel being open: restoring a saved view must apply the + // facet row view even while the panel is collapsed. Callers that only care + // when the panel is visible guard at their own call site. + if (!meta) return; + const { config, dedupContext } = facets; + const dedup = dedupContext?.spec ?? null; + const dedupScope = dedupContext?.scope ?? null; + const active = anyFacetActive(config); + // Claim the newest-sync token before any await. A later sync (a newer + // selection) bumps this again; each await below re-checks it so a stale + // response for an older config can't clobber the newer one. + const token = ++facetSyncSeq; + const superseded = () => facetSyncSeq !== token || get().activeId !== meta.id; + set((s) => ({ facets: { ...s.facets, loading: true, error: null } })); + try { + // 1. Drive the grid row view — but only touch the filter when a facet is + // (or was) active, so merely opening the empty panel never clears a + // filter applied by other means. A view op: bumps the revision, never + // dirties, never enters undo. + let working = meta; + if (active || facets.applied) { + const applied = await api.applyFacets(meta.id, config, meta.revision, dedup, dedupScope); + // A newer selection started syncing while applyFacets was in flight: + // drop this response so it can't apply an older config's row view. + if (superseded()) return; + reloadDoc(applied); + working = applied; + } + // 2. Cross-filtered counts against the (possibly bumped) revision. + const results = await api.computeFacets( + working.id, + config, + working.revision, + dedup, + dedupScope, + ); + if (superseded()) return; // newer sync (or doc switch) owns the state now + set((s) => ({ + facets: { ...s.facets, results, applied: active, loading: false, error: null }, + })); + } catch (e) { + // Only the newest sync owns the loading/error state; a superseded sync + // that errors (e.g. a stale-revision rejection) must stay silent. + if (facetSyncSeq !== token) return; + set((s) => ({ facets: { ...s.facets, loading: false, error: String(e) } })); + } + }, + + copyFacet: async (id) => { + const result = get().facets.results?.facets.find((f) => f.id === id); + if (!result) return; + await writeClipboard(facetClipboardText(result)).catch(() => undefined); + }, + + convertFacetsToFilter: async () => { + const meta = activeMeta(); + const { config } = get().facets; + if (!meta) return; + try { + const conversion = await api.convertFacetsToFilter(meta.id, config); + const summary = summarizeConversion(conversion); + if (summary.empty) { + set({ + error: + summary.dropped.length > 0 + ? `No facet could be converted to a filter. ${describeDropped(summary.dropped)}` + : "There are no active facets to convert.", + }); + return; + } + // Surface the built filter in the existing FilterDialog for review/edit. + const hydrated = hydrateFilter(conversion.filter); + const spec: FilterGroup = { ...hydrated, type: "group", id: hydrated.id ?? "root" }; + set((s) => ({ + filter: { ...s.filter, spec }, + activeModal: "filter", + error: summary.dropped.length > 0 ? describeDropped(summary.dropped) : s.error, + })); + } catch (e) { + set({ error: String(e) }); + } + }, + // ----- file profiles (F08) ------------------------------------------------- saveProfiles: async (profiles) => { diff --git a/src/types.ts b/src/types.ts index c568475..18919f9 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1413,6 +1413,121 @@ export interface NamedView { wrapText: boolean; /** F42: conditional-highlighting rules saved with this view (view-only). */ highlightRules?: HighlightRule[]; + /** F39: multi-facet exploration configuration saved with this view. */ + facets?: FacetConfig | null; +} + +// ----- multi-facet exploration (F39) ---------------------------------------- + +/** The ten facet dimensions. Mirrors `facets::FacetKind`. */ +export type FacetKind = + | "text" + | "number" + | "date" + | "boolean" + | "nullability" + | "semantic" + | "diagnostics" + | "validation" + | "duplicate" + | "annotation"; + +/** Whether selected values keep (include) or remove (exclude) matching rows. */ +export type FacetMode = "include" | "exclude"; + +/** A continuous inclusive range for number / date facets (bounds as strings so + * they parse under the column's declared schema, like a filter range). */ +export interface FacetRange { + min?: string | null; + max?: string | null; +} + +/** One facet's active selection: OR among `values`, plus an optional `range`. */ +export interface FacetSelection { + mode: FacetMode; + values: string[]; + range: FacetRange; +} + +/** One facet panel's full specification (slice + selection + display + layout). + * The array order is the panel order; layout fields round-trip in a saved view. */ +export interface FacetSpec { + id: string; + kind: FacetKind; + columnId?: string | null; + semantic?: SemanticType | null; + selection: FacetSelection; + topN?: number | null; + search?: string | null; + bins?: number | null; + pinned: boolean; + collapsed: boolean; + width?: number | null; +} + +/** The saved multi-facet configuration (extends a named F12 view). */ +export interface FacetConfig { + facets: FacetSpec[]; +} + +/** One computed bucket: a selectable value/category with its cross-filtered count. */ +export interface FacetBucket { + key: string; + label: string; + count: number; + selected: boolean; + /** Numeric/date histogram bins carry their edges (timestamps for dates). */ + lo?: number; + hi?: number; +} + +/** Observed extent and current selection of a number / date facet. */ +export interface RangeInfo { + min: string | null; + max: string | null; + selectedMin: string | null; + selectedMax: string | null; +} + +/** One facet's computed result: its bounded buckets plus metadata. */ +export interface FacetResult { + id: string; + kind: FacetKind; + columnId?: string; + mode: FacetMode; + active: boolean; + /** Column/inputs could not be resolved (missing column / no cached data). */ + unresolved: boolean; + /** Counts estimated from a sample (large indexed document). */ + sampled: boolean; + /** Text facet only: distinct-value map hit its memory cap. */ + truncated: boolean; + /** Text facet only: distinct values observed. */ + distinct?: number; + buckets: FacetBucket[]; + range?: RangeInfo; +} + +/** The full result of a facet computation over one document. */ +export interface FacetResultSet { + revision: number; + matchedRows: number; + totalRows: number; + scannedRows: number; + sampled: boolean; + facets: FacetResult[]; +} + +/** A facet that had no faithful column-filter equivalent during conversion. */ +export interface DroppedFacet { + id: string; + reason: string; +} + +/** Result of the one-way facets → filter-builder conversion. */ +export interface FacetConversion { + filter: FilterGroup; + dropped: DroppedFacet[]; } /** The persisted settings document (versioned JSON in app-data). */