Skip to content

Pluggable Backend interface + FlowLog backend as an experimental crate#1

Closed
oflatt-claude wants to merge 5 commits into
encoding-rebuild-via-orfrom
backend-interface
Closed

Pluggable Backend interface + FlowLog backend as an experimental crate#1
oflatt-claude wants to merge 5 commits into
encoding-rebuild-via-orfrom
backend-interface

Conversation

@oflatt-claude

@oflatt-claude oflatt-claude commented Jul 2, 2026

Copy link
Copy Markdown
Owner

What

Adds a small, documented pluggable backend interface to egglog so a third party can implement their own e-graph execution engine, and demonstrates it with a FlowLog (differential-dataflow) backend living as a crate under egglog-experimental/.

Draft — opened for review. Base is encoding-rebuild-via-or (ff04e0e) so the diff is exactly the 5 backend commits.

Commits

  1. Add a pluggable Backend interface to egglog — new egglog-backend-trait crate; frontend EGraph holds Box<dyn Backend>; EGraph::with_backend(..).
  2. Point egglog-experimental at the local egglog fork — forward-port its API drift to the newer egglog.
  3. Add a FlowLog backend as an egglog-experimental crate — egglog-experimental/flowlog/.
  4. Clippy + rustfmt cleanup.
  5. Wire flowlog into the test matrix — run part of egglog's .egg corpus through flowlog, unconditionally.

Design

  • Clean minimal SPI. The Backend trait's required surface is ~20 methods (tables, rules, iteration via ScanEntry, value registries, primitives). Advanced features (action registry, containers, subsumption, or-patterns) are optional default-provided methods, so an external backend implements only the small core. Ergonomic generics live in a BackendExt blanket impl.
  • egglog-bridge is untouched. The trait crate depends on the bridge and hosts impl Backend for egglog_bridge::EGraph itself (orphan-rule-legal), reusing the bridge's id/config types.
  • FlowLog is a genuine out-of-tree impl depending only on egglog-backend-trait + egglog-core-relations (not the reference bridge), in its own isolated cargo workspace so its heavy differential-dataflow deps don't leak.

FlowLog scope

The Interpret-mode differential-dataflow join (what .egg files actually run on) is fully preserved — rule bodies lower to DD .join chains on a shared timely worker with signed-delta epoch stepping, and an unlowerable rule shape still panics (no host nested-loop fallback). Because the minimal trait's MergeFn/DefaultVal drop the proof-mode / native-merge / term-build / value-fold / native-UF / identity-on-miss variants, that machinery is stripped; merges collapse to the plain Relation/Old/New/Min modes.

Tests

  • egglog: clippy --tests -D warnings ✓, fmt --check ✓, 747 file tests + all unit/integration pass.
  • egglog-experimental: 86 tests pass on the local egglog.
  • flowlog: 7 tests — smoke, commutativity + distributivity through the DD join, and 7 real corpus files (bool/i64/bitwise/interval/before-proofs/merge-saturates/string).

Known pre-existing failure (in the base, not this PR)

eqsat_basic_term_encoding_roundtrip fails on the base commit ff04e0e ("Term encoding: rebuild via correlated OR"), independent of this work. The correlated-or rebuild (rebuilding_rules_correlated in proofs/proof_encoding.rs) emits per-branch throwaway variables l{i}_ that the desugarer re-prefixes with the internal-symbol marker on every pass, so resolve_program is not idempotent for them and text_twice != text_thrice. Fixable by minting those vars with the internal prefix (or as wildcards) so they aren't re-freshened.

Limitations

  • clone_boxed is unimplemented! on flowlog (push/pop unsupported, as in the original).
  • FlowLog supports only the subset above; corpus files needing proofs/containers/subsumption/complex-merges are omitted, and some saturating programs are slow on the DD path.

🤖 Generated with Claude Code

oflatt and others added 5 commits July 2, 2026 19:04
@oflatt @claude
Add a pluggable Backend interface to egglog
34ec838 
Introduce `egglog-backend-trait`, a small, documented SPI so downstream
crates can implement their own e-graph execution backend:

- `Backend`: object-safe trait for table lifecycle, rules, iteration,
  value registries, and primitives. Advanced features (action registry,
  containers, subsumption, `or`-patterns) are optional default-provided
  methods, so an external backend implements only the small required core.
- `BackendExt`: blanket-impl ergonomic sugar (for_each, with_execution_state,
  base_value_constant, register_container_ty) over any `Backend`.
- `RuleBuilderOps`: mirrors the bridge's RuleBuilder; backends accumulate
  into their own IR and finalize on `build()`.
- `impl Backend for egglog_bridge::EGraph` (the in-memory reference backend)
  lives in the trait crate, leaving egglog-bridge untouched.

Route the frontend `EGraph` through `Box<dyn Backend>` and add
`EGraph::with_backend(..)` so a custom backend can be installed. Sort
`column_ty`/`register_type` now take `&BaseValues` / `&mut dyn Backend`.
Bridge-only paths (`unstable-fn` TableAction, scheduler match
instantiation, file loading) downcast via the documented escape hatch.

The full egglog test suite passes (747 file tests + unit/integration),
with no new failures.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oflatt @claude
Point egglog-experimental at the local egglog fork
32d37ce 
Switch experimental's egglog/egglog-ast/egglog-reports deps from the pinned
upstream git rev to the local `../egglog`, per the monorepo's intent, and
forward-port the API drift to the newer egglog:

- FunctionRow (removed) -> reconstruct full rows via a `for_each_full_row`
  helper that tries `function_entries` and falls back to `constructor_enodes`.
- CostModel::enode_cost now takes `&Enode<'_>`; key is `enode.children`.
- lookup_function -> `read(|rs| rs.lookup(name, RawValues(..)))`.
- with_full_state -> `update(|state| { .. ; Ok(()) })`; row reinsertion via
  `state.set` (functions) / `state.add` + `state.union` (constructors).
- GenericRule.naive: bool -> eval_mode: RuleEvalMode (+ include_subsumed).

`cargo build` is clean and all 86 experimental tests pass on the local egglog.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oflatt @claude
Add a FlowLog (differential-dataflow) backend as an egglog-experiment…
55e451f 
…al crate

`egglog-experimental/flowlog/` is a worked example that a third party can
implement their own egglog execution backend against the public
`egglog-backend-trait` SPI — it depends only on that trait plus the neutral
`egglog-core-relations`, not on the reference `egglog-bridge`.

It ports the FlowLog backend (host-side rule iteration with the rule-body join
running on an in-process differential-dataflow dataflow, `dd_native.rs`) onto
the minimal `Backend` trait. Because the minimal trait's `MergeFn`/`DefaultVal`
drop the proof-mode / native-merge / term-build / value-fold / native-UF /
identity-on-miss variants, that machinery is stripped; merges collapse to the
plain Relation/Old/New/Min modes. The Interpret-mode differential-dataflow join
(the real path `.egg` files run on) is fully preserved — rule bodies still
lower to DD `.join` chains on a shared timely worker with signed-delta epoch
stepping, and an unlowerable rule shape still panics (no host nested-loop
fallback).

Its heavy edition-2021 deps (differential-dataflow, timely, flowlog-rs,
vendored differential-dogs3) are isolated in the crate's own workspace so they
don't leak into the egglog workspace.

Tests: `smoke.rs` runs a datatype + insertion + `(run 1)` + `(print-size)`
through `EGraph::with_backend(FlowlogEGraph)`; `rewrite_join.rs` confirms
commutativity (single-atom join) and distributivity (multi-atom join) rewrites
fire through the DD join and unify via congruence.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oflatt @claude
Clippy + rustfmt cleanup across the backend-interface work
93740b3 
- egglog-backend-trait: drop redundant `|e| f(e)` closures in the bridge impl.
- egglog: remove a now-useless `QueryEntry -> QueryEntry` `.into()` in the rule
  translator (lookup/call return QueryEntry directly through the trait).
- rustfmt across the touched egglog and flowlog files.

`cargo clippy --tests -- -D warnings` and `cargo fmt --check` pass on the egglog
workspace; egglog / experimental / flowlog test suites are green (the only
failing test, eqsat_basic_term_encoding_roundtrip, is pre-existing on the base).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oflatt @claude
Wire flowlog into the test matrix: differential test vs the reference…
b234e03 
… backend

Add `flowlog/tests/corpus.rs`, which runs a curated subset of egglog's own
`.egg` corpus on BOTH the reference in-memory backend and the FlowLog backend
and asserts they agree — unconditionally (no env gating), as part of
`cargo test`.

Both egraphs are built identically (`EGraph::default()` vs
`EGraph::with_backend(FlowlogEGraph)`), so the frontend drives the same
operations into each backend. Every file gets `(print-size)` appended and the
two backends' per-function row-count output is compared: if FlowLog's
differential-dataflow engine derives a different final database than the
reference, the counts diverge and the test fails. All listed files
(bool/i64/bitwise/interval/before-proofs/merge-saturates/string) currently
match the reference exactly.

Files needing features outside FlowLog's subset (proofs, containers,
subsumption, complex merges, non-DD-lowerable rules) are omitted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oflatt-claude

oflatt-claude commented Jul 2, 2026

Copy link
Copy Markdown
Owner Author

Superseded by saulshanabrook#4 (opened against the upstream repo).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@oflatt-claude @oflatt