feat(f38): data dictionary and schema documentation

CHANGELOG.md

@@ -63,6 +63,24 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
   dialog lists each source's status with per-source actions (open, relink,
   leave out, remove, or open-available-only); and quitting or closing a
   project with unsaved workspace changes prompts to save first.
+- **Data dictionary** — document what each column MEANS. Every column
+  carries an optional entry (display name, description, analytical role,
+  unit, source, sensitivity, allowed values, example, owner, notes) keyed
+  by its stable column ID, so the documentation survives renames and
+  reorders and is restored by undo/redo; deleting a column reports its
+  entry as orphaned and keeps it until you explicitly discard it (an undo
+  re-attaches it). Editing the dictionary is pure metadata: it has its own
+  revision, like the schema, and never rewrites a cell or marks the
+  document dirty. The editor prefills each column's technical name and
+  inferred F31 type. Dictionaries import and export as versioned CEESVEE
+  JSON, Markdown documentation, and tabular CSV documentation; an import
+  merges incoming metadata by column ID (or by mapped column name when the
+  IDs differ) and surfaces every field-level conflict for explicit
+  resolution before it replaces anything. File profiles can require
+  documentation fields (e.g. a description and owner on every column) as
+  ordinary validation issues, and columns classified confidential or
+  restricted are folded into the PII scan preflight even when no detector
+  matches them.
 - **Explicit schemas and typed columns** (palette → "Edit schema…", or a
   column header's menu): declare an explicit logical type per column —
   text, integer, decimal, float, boolean, date, datetime, UUID, or JSON —

README.md

@@ -162,6 +162,20 @@ and faithful on large, real-world delimited files.**
   conversion applies as one previewed, undoable step. Schemas key columns
   by stable IDs so they survive renames and reorders; a violet header
   badge marks a declared type.
+- **Data dictionary** — document what each column MEANS: display name,
+  description, analytical role, unit, source, sensitivity, allowed values,
+  example, owner, and notes, each keyed by the stable column ID so the
+  documentation survives renames and reorders (deleting a column reports
+  its entry as orphaned and keeps it). The searchable editor prefills every
+  column's technical name and inferred type, shows a per-column
+  completeness indicator, and surfaces the description, unit, and a
+  sensitivity badge as a column-header tooltip. Editing the dictionary is
+  pure metadata — it has its own revision and never dirties the document.
+  Import and export as versioned CEESVEE JSON, Markdown, or CSV
+  documentation; an import merges by column ID (or mapped name) and every
+  field-level conflict is resolved explicitly before it replaces anything.
+  File profiles can require documentation fields, and columns marked
+  confidential or restricted feed the personal-data preflight.
 - **Cross-column validation** — relational rules between columns (typed
   comparisons, date order, conditional required, sum equality with
   tolerance, allowed combinations, …) with violation samples, jump-to-row,

src-tauri/src/commands.rs

@@ -21,6 +21,10 @@ use crate::crossval::{self, CrossRule, CrossValCache, CrossValReport};
 use crate::dedup::{self, DedupCache, DedupSpec, DuplicateKeepStrategy, DuplicateReport};
 use crate::diagnostics::{self, DiagnosticsCache, DiagnosticsReport};
 use crate::dialect::{self, CsvDialectOptions, DialectPreview};
+use crate::dictionary::{
+    self, DictionaryField, DictionaryFormat, DictionaryImportOutcome, DictionaryView, MergeMatchBy,
+    MergePlan, MergeResolution,
+};
 use crate::document::{ChangeSummary, Document};
 use crate::dto::{
     BackupPolicy, CellRect, ColumnSummary, DocumentMeta, EncodingCompatibility,
@@ -582,8 +586,10 @@ pub async fn start_reindex(
             fresh.set_revision(doc.revision() + 1);
             fresh.set_fingerprint(fingerprint);
             // Schema entries key on stable IDs, which restart positionally on
-            // a reload — they re-attach to the same columns (F31).
+            // a reload — they re-attach to the same columns (F31). The data
+            // dictionary (F38) carries across on the same principle.
             fresh.inherit_schema(&doc);
+            fresh.inherit_dictionary(&doc);
             *doc = fresh;
             Ok(())
         })
@@ -666,8 +672,10 @@ pub async fn apply_reparse(
         fresh.set_revision(doc.revision() + 1);
         fresh.set_fingerprint(fingerprint);
         // Schema entries key on stable IDs, which restart positionally on a
-        // reparse — they re-attach to the same columns (F31).
+        // reparse — they re-attach to the same columns (F31). The data
+        // dictionary (F38) carries across on the same principle.
         fresh.inherit_schema(doc);
+        fresh.inherit_dictionary(doc);
         // Journaling continues against the NEW interpretation.
         attach_journal_if_enabled(&app, &mut fresh);
         let meta = fresh.meta();
@@ -1886,6 +1894,166 @@ pub async fn apply_redaction(
     Ok(meta)
 }
 
+// ----- data dictionary (F38) ----------------------------------------------------
+
+/// The dictionary editor surface: one row per current column (technical name +
+/// inferred F31 type prefilled, stored entry when documented) plus any orphaned
+/// entries. `dictionaryRevision` is the metadata revision used to guard edits;
+/// documentation edits never move the document `revision` or the dirty flag.
+#[tauri::command]
+pub fn get_dictionary(doc_id: u64, state: Db<'_>) -> AppResult<DictionaryView> {
+    read_doc(&state, doc_id, |doc| Ok(dictionary::view(doc)))
+}
+
+/// Insert or replace one column's documentation. An entry with no populated
+/// field is removed rather than stored empty. Metadata only: not undoable,
+/// never dirties the document. Guarded by the dictionary revision.
+#[tauri::command]
+pub fn set_dictionary_field(
+    doc_id: u64,
+    field: DictionaryField,
+    expected_dictionary_revision: u64,
+    state: Db<'_>,
+) -> AppResult<DictionaryView> {
+    write_doc(&state, doc_id, |doc| {
+        doc.check_dictionary_revision(expected_dictionary_revision)?;
+        dictionary::validate_field(&field)?;
+        // The entry must key on a column that exists (present or orphaned is
+        // fine — it is keyed by stable ID either way).
+        if field.is_documented() {
+            doc.set_dictionary_field(field);
+        } else {
+            doc.remove_dictionary_field(&field.column_id);
+        }
+        Ok(dictionary::view(doc))
+    })
+}
+
+/// Drop one column's documentation entry (clearing a column, or discarding a
+/// single orphan). Guarded by the dictionary revision.
+#[tauri::command]
+pub fn remove_dictionary_field(
+    doc_id: u64,
+    column_id: String,
+    expected_dictionary_revision: u64,
+    state: Db<'_>,
+) -> AppResult<DictionaryView> {
+    write_doc(&state, doc_id, |doc| {
+        doc.check_dictionary_revision(expected_dictionary_revision)?;
+        doc.remove_dictionary_field(&column_id);
+        Ok(dictionary::view(doc))
+    })
+}
+
+/// Discard EVERY orphaned entry (documentation whose column is gone). The
+/// user's explicit "clean up orphans" action. Guarded by the dictionary
+/// revision.
+#[tauri::command]
+pub fn discard_dictionary_orphans(
+    doc_id: u64,
+    expected_dictionary_revision: u64,
+    state: Db<'_>,
+) -> AppResult<DictionaryView> {
+    write_doc(&state, doc_id, |doc| {
+        doc.check_dictionary_revision(expected_dictionary_revision)?;
+        let orphan_ids: Vec<String> = dictionary::orphans(doc)
+            .into_iter()
+            .map(|o| o.column_id)
+            .collect();
+        if !orphan_ids.is_empty() {
+            let mut dict = doc.dictionary().clone();
+            for id in &orphan_ids {
+                dict.remove(id);
+            }
+            doc.set_dictionary(dict);
+        }
+        Ok(dictionary::view(doc))
+    })
+}
+
+/// Export the dictionary as versioned JSON, Markdown documentation, or tabular
+/// CSV documentation (atomic write via the F03 pipeline).
+#[tauri::command]
+pub async fn export_dictionary(
+    doc_id: u64,
+    path: String,
+    format: DictionaryFormat,
+    state: Db<'_>,
+) -> AppResult<()> {
+    let handle = doc_handle(&state, doc_id)?;
+    tauri::async_runtime::spawn_blocking(move || {
+        let rendered = {
+            let doc = handle.read().map_err(poisoned)?;
+            dictionary::export_as(&doc, format)?
+        };
+        save_mod::atomic_write(Path::new(&path), BackupPolicy::None, |file| {
+            use std::io::Write;
+            file.write_all(rendered.as_bytes())?;
+            Ok(rendered.len() as u64)
+        })?;
+        Ok(())
+    })
+    .await
+    .map_err(|e| AppError::Other(format!("background task failed: {e}")))?
+}
+
+/// Plan a dictionary import: parse the CEESVEE dictionary JSON at `path`, match
+/// its entries to current columns by ID or mapped name, and return the merge
+/// plan (clean additions + the field-level conflicts that must be resolved
+/// before applying). Read-only — nothing changes.
+#[tauri::command]
+pub async fn preview_dictionary_import(
+    doc_id: u64,
+    path: String,
+    match_by: MergeMatchBy,
+    state: Db<'_>,
+) -> AppResult<MergePlan> {
+    let handle = doc_handle(&state, doc_id)?;
+    tauri::async_runtime::spawn_blocking(move || {
+        let json = std::fs::read_to_string(&path)?;
+        let imported = dictionary::parse_import(&json)?;
+        let doc = handle.read().map_err(poisoned)?;
+        Ok(dictionary::plan_merge(&doc, &imported, match_by))
+    })
+    .await
+    .map_err(|e| AppError::Other(format!("background task failed: {e}")))?
+}
+
+/// Apply a dictionary import under an explicit conflict resolution. Fails
+/// (changing nothing) if any reported conflict is left unresolved, or if the
+/// dictionary moved since the plan was taken. Metadata only: never dirties the
+/// document.
+#[tauri::command]
+pub async fn apply_dictionary_import(
+    doc_id: u64,
+    path: String,
+    match_by: MergeMatchBy,
+    resolution: MergeResolution,
+    expected_dictionary_revision: u64,
+    state: Db<'_>,
+) -> AppResult<DictionaryImportOutcome> {
+    let handle = doc_handle(&state, doc_id)?;
+    tauri::async_runtime::spawn_blocking(move || {
+        let json = std::fs::read_to_string(&path)?;
+        let imported = dictionary::parse_import(&json)?;
+        let mut doc = handle.write().map_err(poisoned)?;
+        doc.check_dictionary_revision(expected_dictionary_revision)?;
+        let applied = dictionary::apply_merge(&doc, &imported, match_by, &resolution)?;
+        doc.set_dictionary(applied.dictionary);
+        Ok(DictionaryImportOutcome {
+            matched_columns: applied.matched_columns,
+            new_entries: applied.new_entries,
+            updated_entries: applied.updated_entries,
+            fields_added: applied.fields_added,
+            conflicts_resolved: applied.conflicts_resolved,
+            unmatched: applied.unmatched,
+            view: dictionary::view(&doc),
+        })
+    })
+    .await
+    .map_err(|e| AppError::Other(format!("background task failed: {e}")))?
+}
+
 // ----- batch recipes (F25) ------------------------------------------------------
 
 /// Validate a batch (recipe version, steps, templates, distinct output