RFC: Offline memory consolidation ("auto-dream") — idle-time cluster / abstract / supersede

[dcfocus]

Summary

RFC for an offline memory consolidation capability ("auto-dream"): an opt-in background / idle-time process that clusters related ContextRecords, abstracts them into higher-level memories, links the result to its sources, and retires the redundant raw records — producing more compact, higher-signal online memory over time. Inspired by Generative-Agents "reflection" and the 2025 "sleep-time compute" line.

Storage-native machinery only. The abstraction step is a user-supplied hook; lance-context bundles no models.

Motivation

Long-running agents accumulate large volumes of low-level episodic memory. As the store grows with redundant / near-duplicate entries, recall quality and cost degrade. Consolidation — summarize many raw turns into a few semantic memories, drop redundancy — keeps memory compact and high-signal. lance-context already holds the raw memory and has the primitives to do this in place, versioned and auditable.

Why here (the bet)

Consolidation maps onto primitives lance-context already has:

• embeddings → cluster related / near-duplicate memories
• relationships → link a derived memory back to the source records it summarizes
• lifecycle (supersede / retire) → replace consolidated-away raw records non-destructively (auditable, reversible)
• versioning → each consolidation pass is a new version you can inspect or roll back
• background compaction infra (should_compact, store.rs:1209; start_background_compaction, store.rs:1284) → an existing pattern for scheduling bounded background work

Proposed shape (to discuss)

1. Candidate selection — cluster by embedding similarity, or group by session / run / time window, to find redundant or related sets.
2. Abstraction hook (user-supplied) — given a cluster of records, the user's callback returns a consolidated record (summary / abstraction). lance-context invokes the hook and persists the result; it bundles no models.
3. Write-back — insert derived records (source="consolidated"), link via relationships, supersede / retire the originals, bump a version.
4. Scheduling — opt-in background pass (idle / cadence / threshold-triggered), reusing the compaction scheduler pattern; bounded and cancellable.
5. Safety / reproducibility — never hard-delete (lifecycle only); record consolidation params + source ids; version-pinned and reversible.

Relationship to #96 (important)

Auto-dream and #96 curation are the same core loop — read records → derive new records → supersede originals → version. They differ only in:

• trigger — background / idle (auto-dream) vs. on-demand (curation)
• output — better online memory (auto-dream) vs. trainable export (curation)

This RFC should decide whether to build one shared "derive-and-supersede" engine with two front-ends (consolidation + curation/export) rather than two parallel implementations.

Non-goals

• Not an agent framework: bundles no LLM / prompting; abstraction is a hook.
• No hard deletion — consolidation is lifecycle-based and reversible.
• Not a replacement for compaction (feat: add background compaction for Lance fragments #16): that is physical file optimization; this is semantic consolidation.

Open questions

• Demand — is there a concrete workflow that needs in-store consolidation rather than doing it in the agent layer? (Same validation bar we applied to Curate lance-context records into trainable datasets (SFT/preference/rollout export) #96.)
• Clustering approach and thresholds; how to avoid lossy over-abstraction.
• Interaction with search — do consolidated memories rank over raw? do raw records stay searchable, or only the abstraction?
• Shared derive-and-supersede engine with Curate lance-context records into trainable datasets (SFT/preference/rollout export) #96 — yes or no?

Acceptance (RFC stage)

• Agreement on scope / non-goals (machinery + hook, no bundled models), the shared-engine decision with Curate lance-context records into trainable datasets (SFT/preference/rollout export) #96, and a short design doc under docs/design/ (mirrors how Explore partitioned namespace layout for scoped agent context #90 was handled). Implementation tracked in follow-ups.

[dcfocus]

Now that #96 (curate + export) has landed its first slice, I can answer the central open question here — the shared "derive-and-supersede" engine — concretely, plus the other open questions.

Scope / non-goals: +1

Agree with the framing: storage-native machinery + a user-supplied abstraction hook, no bundled models. Consolidation is lifecycle-based (supersede/retire, never hard-delete), versioned, and reversible. That keeps this distinct from physical compaction (#16) and from being an agent framework.

Shared engine with #96: yes — but be precise about what's shared

#96 and auto-dream are the same loop on the read side and diverge on the sink side. Concretely, #96's export module already implements:

• candidate selection — list_filtered_with_options (lifecycle-correct: drops tombstoned/expired/retired/superseded/contradicted) + metadata/quality filters;
• clustering / redundancy — cosine dedup and decontamination helpers already operate on embeddings;
• grouping/aggregation — by session/run/tenant/external-id prefix, deterministically ordered;
• provenance + manifest — source record ids, version, params.

What auto-dream adds that export deliberately omits is a write-back sink:

• insert derived record(s) with source="consolidated",
• link them to sources via relationships,
• supersede/retire the originals (already supported via upsert/update lifecycle),
• bump a version.

So the right factoring is one selection/clustering substrate, two sinks:

select (filters + lifecycle + cluster)  ──► transform (user hook)  ──► sink
                                                                      ├─ export:        JSONL + manifest      (#96, done)
                                                                      └─ consolidate:   insert + link + supersede + version  (#99)

I'd extract the candidate-selection + cosine-cluster + grouping helpers out of export.rs into a small shared curate/cluster module, then build consolidate() on top of it rather than a parallel implementation. The transform hook differs only in signature: export formats a group to a training row; consolidation maps a cluster to a new ContextRecord.

Open questions

Demand. Hold to the same bar we applied to #96: ship the machinery as an opt-in API, but do not enable a background scheduler by default until at least one concrete agent workflow needs in-store (vs. agent-layer) consolidation. The synchronous consolidate(options, hook) call is useful on its own (cron/idle from the caller) and de-risks the design before we invest in the scheduler.

Clustering & thresholds (avoiding lossy over-abstraction). Reuse the cosine distance already in export.rs; greedy near-duplicate collapse for v1 with an exposed similarity floor + max_cluster_size. Guardrails:

• only consolidate clusters above a similarity floor and a minimum size (don't summarize singletons);
• never abstract across session_id/tenant boundaries by default (configurable), to avoid blending unrelated memory;
• originals are retired, not deleted, so every pass is reversible and auditable via the manifest.

Interaction with search. Default: consolidated (L0) records are searchable; superseded/retired originals drop out of default recall but remain in the store. Traceability is already available through LifecycleQueryOptions { include_retired: true } + relationships, which is exactly the L0→L1→L2 trace-back described in #37. So raw memory stays queryable for evidence; it just stops competing in normal recall.

Shared engine — yes (above): shared select/cluster, separate sink.

Proposed next steps (RFC acceptance)

1. Land a short docs/design/auto-dream.md capturing: scope/non-goals, the shared select→transform→sink factoring, lifecycle/reversibility model, default guardrails, and the scheduler-deferred decision (mirrors how Explore partitioned namespace layout for scoped agent context #90 was handled). I can open this PR.
2. Refactor the candidate-selection + cosine-cluster + grouping helpers from export.rs into a shared module (pure refactor, no behavior change).
3. Implement opt-in consolidate(ConsolidateOptions, hook) -> ConsolidationManifest (insert + relationship-link + supersede + version), reusing (2).
4. Defer the background/idle scheduler (reusing the should_compact / start_background_compaction pattern) to a follow-up gated on validated demand.

If this direction has agreement, I'll start with the design doc in (1) so the shared-engine decision is recorded before any implementation. Sound good?

[dcfocus]

Started the design doc (step 1 from the plan above) in #114 — docs/design/auto-dream.md, mirroring partitioned-namespace.md. It records scope/non-goals, the shared select→transform→sink engine with #96 (extract selection/cluster/group from export.rs; export and consolidate as two sinks), search/traceability via retire + consolidates relationships, over-abstraction guardrails, and flags the one new storage primitive needed (atomic N→1 supersede). Scheduler deferred behind a synchronous consolidate(options, hook).

It's a draft for the RFC discussion — happy to iterate on the branch as you weigh in, especially on the shared-engine call and the N→1 supersede shape.