feat(dedup): opt-in near-duplicate observation dedup — Tier-0 auto-merge + Tier-1 candidates (#3038)

[crippledgeek]

Refs #3038.

Opt-in, off-by-default near-duplicate dedup for observations. content_hash only
catches byte-identical rows, so paraphrased observations of the same recurring work
accumulate as noise. This adds two deterministic tiers, validated against a real
7,651-observation DB before any code was written (see the diagnostic on #3038).

This is the conservative interpretation of #3038 (exact auto-merge + fuzzy
candidate-flagging, not fuzzy auto-merge). The data showed fuzzy lexical
auto-merge is unsafe — happy to switch to auto-merge if you'd prefer; I asked on
the issue. The LLM-adjudication tier that could safely resolve the residual cases
is deliberately deferred to a follow-up PR.

What it does (all gated on CLAUDE_MEM_DEDUP_ENABLED, default false)

• Tier 0 — exact-normalized-title → silent auto-merge. A new observation whose title
  matches an existing one after lowercase/whitespace-collapse/punctuation-strip is collapsed
  onto the existing row (cross-session) and its new occurrence_count is bumped. O(1) via an
  indexed title_norm_key (sha256(project + normalizeTitle)), mirroring the content_hash
  pattern. NULL key for empty/emoji/punctuation-only titles so they never collapse into
  each other.
• Tier 1 — IDF-weighted TF-IDF cosine + an IDF "veto" → review-only candidates. Records
  near-dup candidates in observation_dedup_candidates (never auto-merged). The veto
  (a Fellegi–Sunter blocking key) rejects pairs that differ only in a rare discriminating
  token — rdlp-api vs rdlp-plugin, ffmpeg-7.1 vs 6.1 — which plain string similarity
  wrongly scores ~0.9.

Why this shape (validation evidence)

On the real DB at production thresholds: SimHash-over-narrative was useless (false-positive
dominated); the signal lives in the title; exact-normalized-title found 35–47 real
redundancies content_hash misses with zero false positives; and fuzzy lexical auto-merge
would have destroyed genuinely-distinct work that differs in one token — hence Tier-1 is
review-only. Semantic paraphrases (different words, same meaning) need embeddings and are out
of scope here.

Surfaces

• Migration v33 (pure SQL, idempotent): occurrence_count, token_df (IDF model),
  dedup_meta, observation_dedup_candidates.
• POST /api/dedup/scan — opt-in idempotent backfill (so existing DBs participate) + bounded
  inverted-index corpus sweep. Gated on the flag + single-flight + row-cap.
• GET /api/dedup/candidates?project= — read-only candidate list.
• 7 CLAUDE_MEM_DEDUP_* settings (documented in configuration.mdx).

Disabled ⇒ byte-identical legacy behavior (regression-tested).

Tests

Strict TDD, ~20 commits. 175 dedup/sqlite/settings tests green; tsc --noEmit clean; full
bun test green. Coverage includes the empty-title data-loss guard, cross-project isolation,
cross-session + intra-batch Tier-0, cold-start gate, retry-idempotency (token_df/doc_count stay
flat on a merge or redelivery), and a real-DB-derived golden fixture.

Reviews

Both ran over the full BASE..HEAD diff, fixes applied, then re-reviewed:

• security-reviewer → PASS. No injection / auth bypass / data-loss. Initial MEDIUM findings
  (scan concurrency guard; prepared-statement hardening) + LOWs (backfill row cap; trust-model
  doc) all resolved and re-confirmed.
• code-reviewer → Approve. Initial findings (occurrence_count retry-idempotency, a
  token_df-flat regression test, scan-when-disabled gating, IDF single-load perf, NaN-config
  clamp, DRY) all resolved with a genuine regression test; re-confirmed merge-ready.

Trust model

Like all worker routes, /api/dedup/* is guarded only by the 127.0.0.1 binding + localhost
CORS (consistent with DataRoutes/SearchRoutes/MemoryRoutes).

Follow-up (separate PR, only if this is accepted)

An async LLM-adjudication consolidation tier (with a tombstone/audit subsystem) to resolve the
common-token cases lexical methods can't (Code vs Security review) — kept out of this PR
because a wrong auto-merge in an append-only store is unrecoverable, so it needs its own
reversibility machinery.

[greptile-apps]

Greptile Summary

Adds an opt-in (default-off) two-tier near-duplicate dedup for observations: Tier-0 performs silent auto-merge on exact-normalized-title matches (cross-session), and Tier-1 records IDF-cosine near-duplicates as review-only candidates without ever auto-merging. The entire feature is gated on CLAUDE_MEM_DEDUP_ENABLED=true, so disabled installs remain byte-identical to prior behavior.

• Tier-0: computeTitleNormKey (SHA-256 of project + normalizedTitle, truncated to 128-bit hex) is stored in an indexed column; on insert, findTier0Canonical finds an existing row in O(1) and bumps its occurrence_count instead of inserting a duplicate.
• Tier-1: Per-insert candidate scan uses IDF-weighted cosine similarity plus an IDF-veto (Fellegi–Sunter blocking key) to flag near-duplicate pairs into observation_dedup_candidates for human review; the full-corpus sweep at POST /api/dedup/scan backfills both the IDF model and candidates for existing databases.
• Schema migration v36: New columns (occurrence_count, title_norm_key) on observations, plus three new tables (token_df, dedup_meta, observation_dedup_candidates); all DDL is idempotent and guarded by CREATE TABLE IF NOT EXISTS / column-existence checks.

Confidence Score: 5/5

Safe to merge; the feature is off by default and all dedup paths are gated behind CLAUDE_MEM_DEDUP_ENABLED, leaving disabled installs untouched.

All dedup logic is opt-in, the migration DDL is idempotent, the IDF model and candidate writes are review-only (no auto-merge beyond exact-normalized-title), and the test suite covers the critical data-loss guards. The two findings are a minor strict-vs-non-strict threshold discrepancy and a prepared-statement efficiency concern, neither of which affects correctness or the disabled code path.

src/services/dedup/idfVeto.ts (strict > vs >= threshold boundary) and src/services/sqlite/dedup-store.ts / SessionStore.ts (ephemeral prepared statements in hot paths when dedup is enabled).

Important Files Changed

Filename	Overview
src/services/sqlite/dedup-store.ts	New file implementing IDF model maintenance, Tier-0 canonical lookup, and Tier-1 candidate sweep; generally correct, but bumpTokenDf creates a fresh prepared statement on every call
src/services/sqlite/SessionStore.ts	Adds Tier-0/Tier-1 dedup logic to storeObservation and storeObservations; logic is correct, but inline db.prepare() calls in hot paths (retry check and Tier-0 UPDATE) create ephemeral prepared statements on every invocation
src/services/dedup/idfVeto.ts	IDF-veto uses strict > comparison against thetaIdf, making the effective threshold df < N rather than df ≤ N as documented; boundary token is not vetoed, which is a minor docs/code discrepancy
src/services/dedup/nearDuplicate.ts	Clean Tier-0/Tier-1 classification logic; empty-title guard, sparse-vector noise filter, and veto integration all look correct
src/services/dedup/tfidfCosine.ts	Correct IDF-weighted cosine with zero-norm guard; normA/normB guard prevents NaN on empty inputs
src/services/dedup/normalize.ts	Two deliberately-different normalizers; Unicode-aware strip for Tier-0, whitespace-only split for Tier-1 compound-identifier preservation; both handle null/undefined gracefully
src/services/worker/http/routes/DedupRoutes.ts	Thin route wrapper with correct single-flight guard (static flag + try/finally), feature-flag check on scan, and parameterized queries; consistent with existing route trust model
src/services/sqlite/schema.sql	Adds occurrence_count, title_norm_key, token_df, dedup_meta, and observation_dedup_candidates; correct CHECK constraints, UNIQUE constraint, and CASCADE deletes
src/services/dedup/idf.ts	Smoothed IDF formula handles df=0 correctly; buildIdfFn is a clean closure factory
src/shared/SettingsDefaultsManager.ts	Adds 7 new DEDUP settings with well-chosen defaults; NaN/fractional config clamped in SessionStore.dedupConfig()

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[storeObservation called] --> B{dedup.enabled?}
    B -- No --> G[INSERT with ON CONFLICT DO NOTHING]
    B -- Yes --> C{Same session + content_hash exists?}
    C -- Yes: retry --> R[Return existing row\nno occurrence_count bump]
    C -- No --> D{titleNormKey is null?\ne.g. emoji/punctuation-only}
    D -- Yes --> G
    D -- No --> E{findTier0Canonical:\nsame project + titleNormKey?}
    E -- Yes: Tier-0 merge --> F[UPDATE occurrence_count++\nReturn canonical row\nNo IDF model update]
    E -- No --> G
    G --> H{INSERT returned a row?}
    H -- Yes: real insert --> I[maintainDedupOnInsert:\nbumpTokenDf\ndoc_count++]
    I --> J{isFuzzyReady?\ndoc_count >= minProjectDocs?}
    J -- No: cold start --> K[Return new row id]
    J -- Yes --> L[recordTier1Candidates:\nbuildProjectIdf\nclassify against last maxScan rows]
    L --> M{classifyPair = candidate\nAND IDF-veto does NOT fire?}
    M -- Yes --> N[INSERT OR IGNORE into\nobservation_dedup_candidates\nreview-only, never auto-merged]
    M -- No --> K
    N --> K
    H -- No: ON CONFLICT --> O[Fetch existing by content_hash\nReturn it]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[storeObservation called] --> B{dedup.enabled?}
    B -- No --> G[INSERT with ON CONFLICT DO NOTHING]
    B -- Yes --> C{Same session + content_hash exists?}
    C -- Yes: retry --> R[Return existing row\nno occurrence_count bump]
    C -- No --> D{titleNormKey is null?\ne.g. emoji/punctuation-only}
    D -- Yes --> G
    D -- No --> E{findTier0Canonical:\nsame project + titleNormKey?}
    E -- Yes: Tier-0 merge --> F[UPDATE occurrence_count++\nReturn canonical row\nNo IDF model update]
    E -- No --> G
    G --> H{INSERT returned a row?}
    H -- Yes: real insert --> I[maintainDedupOnInsert:\nbumpTokenDf\ndoc_count++]
    I --> J{isFuzzyReady?\ndoc_count >= minProjectDocs?}
    J -- No: cold start --> K[Return new row id]
    J -- Yes --> L[recordTier1Candidates:\nbuildProjectIdf\nclassify against last maxScan rows]
    L --> M{classifyPair = candidate\nAND IDF-veto does NOT fire?}
    M -- Yes --> N[INSERT OR IGNORE into\nobservation_dedup_candidates\nreview-only, never auto-merged]
    M -- No --> K
    N --> K
    H -- No: ON CONFLICT --> O[Fetch existing by content_hash\nReturn it]

Loading

_{Reviews (3): Last reviewed commit: "fix(dedup): count only newly-persisted c..." | Re-trigger Greptile}

[crippledgeek]

Thanks @greptile-apps — good catch, fixed in 4c67a64.

recordTier1Candidates / sweepProjectCandidates were doing count++ on every INSERT OR IGNORE that classified as a candidate, so a re-run or redelivery reported a phantom count for rows that the UNIQUE(observation_id, duplicate_of_id) guard actually ignored. They now accumulate .run().changes instead, so the returned count reflects only newly-persisted rows — a second idempotent sweep correctly returns 0 (the persisted data was always idempotent; only the reported number was off).

Tightened the idempotency test to assert the return value (not just the row count) — it fails against the old code and passes now. 140 dedup/sqlite tests green, typecheck clean.

[greptile-apps]

The fix is correct and well-executed. The key change in dedup-store.ts:

// Before (buggy):
count++; // incremented for every classifyPair 'candidate', including INSERT OR IGNORE no-ops

// After (correct):
count += ins.run(...).changes; // only increments when the row was actually inserted (changes=1), 0 on UNIQUE conflict

And the test tightening is exactly right — expect(secondReturned).toBe(0) now fails against the old code and proves the fix. The assertion that firstReturned === after1 (return value equals row count on first run) is also a nice addition that covers the forward case.

LGTM. The fix is minimal, precise, and the test proves both the positive and idempotent paths. No concerns.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.