fix: avoid O(matches×graph) re-MATCH in edge search#1500
Conversation
The FalkorDB driver's edge_fulltext_search and edge_bfs_search both
re-MATCH each relationship returned by an index/path call:
CALL db.idx.fulltext.queryRelationships(...) YIELD relationship AS rel
MATCH (n:Entity)-[e:RELATES_TO {uuid: rel.uuid}]->(m:Entity)
FalkorDB's planner does not use the RELATES_TO uuid index to anchor a
relationship inside a node-rel-node pattern, so each yielded row triggers
a graph-wide scan. On a moderate graph (~19k RELATES_TO edges) a single
search reached 25 minutes and pinned multiple cores.
The fulltext/path call already returns the relationship object, so the
endpoints are available directly via startNode()/endNode() — no re-MATCH
needed. The same fix applies to both functions.
Measured on a live instance (2,852 entities / 18,849 RELATES_TO edges):
- edge_fulltext_search: ~25 min -> ~12 ms
- edge_bfs_search (d=2): >45 s -> ~1 s
edge_bfs_search additionally gains correctness: the previous undirected
re-MATCH `(n)-[e {uuid}]-(m)` returned each edge twice with swapped
source/target; startNode()/endNode() yields the true orientation once.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`edge_fulltext_search` in search_utils.py — the live search path used by
`Graphiti.search()` — re-MATCHes every relationship the fulltext index
call already yielded:
CALL db.idx.fulltext.queryRelationships(...) YIELD relationship AS rel
MATCH (n:Entity)-[e:RELATES_TO {uuid: rel.uuid}]->(m:Entity)
The planner cannot use the RELATES_TO uuid index to anchor a relationship
inside a node-rel-node pattern, so each yielded row triggers a graph-wide
scan. On a ~19k-edge FalkorDB graph a single search reached ~25 minutes.
The fulltext call already returns the relationship, so the endpoints are
available via startNode()/endNode() — no re-MATCH. ~25 min -> <1 s,
verified end-to-end against a live FalkorDB instance.
This is the same fix as the earlier commit applied to the FalkorDB
driver's search_ops.py; this commit covers the generic search_utils.py
path, which is the one actually executed when `driver.search_interface`
is unset (the default).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DB-free query-shape regression test asserting edge_fulltext_search and edge_bfs_search derive endpoints via startNode/endNode instead of the O(matches×graph) re-MATCH by uuid. Placed at tests/ root (not tests/driver/, which the unit-test workflow ignores) so it runs in CI; falkordb is installed there via `uv sync --all-extras`. Harvested from getzep#1494 (credit @mturac) and adapted to this PR's BFS rewrite (type(e) guard after rebinding rel -> e). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
927464a to
65653e8
Compare
|
I have read the CLA Document and I hereby sign the CLA behalf on myself, e-mail: example@example.com or I have read the CLA Document and I hereby sign the CLA behalf of my company, e-mail: example@example.com Signature is valid for 6 months. This bot will be retriggered when the Contributor License Agreement comment has been provided. Posted by the CLA Assistant Lite bot. |
|
Maintainer note: rebased onto current main and added a DB-free query-shape regression test ( |
|
Production data in support of merging this: we run graphiti-core 0.28.2 on FalkorDB (self-hosted memory service, one graph per group_id) and the re-MATCH pattern this PR removes was our single biggest failure source — dense-document episodes took 15–25 minutes or dead-lettered on Evidence gathered on 2026-07-14 (graph with ~2.6k Entity nodes):
We can also confirm the maintainer note above from runtime observation: One gap we found while validating: the BFS variant of the same antipattern in |
|
@jaredloman The fulltext half checks out with identical results. But the public edge_bfs_search generic branch still re-MATCHes, your BFS rewrite landed only in the unused FalkorSearchOperations path. Moving it to search_utils.py would complete it. |
|
Independent production confirmation of both the bug and this fix, plus a data point on the BFS residual @Naseem77 flagged. Setup: graphiti-core 0.29.2, FalkorDB, graph of ~10.4k entity nodes / ~7.7k Before: single After applying this PR's On the BFS residual: confirmed — the generic UNWIND relationships(path) AS rel
WITH DISTINCT rel AS e, startNode(rel) AS n, endNode(rel) AS m
WHERE type(e) = 'RELATES_TO'(and any existing |
Problem
Edge search re-MATCHes every relationship that a fulltext / path call already yielded, instead of using the relationship object directly:
The planner can't use the
RELATES_TO.uuidindex to anchor a relationship inside a(node)-[rel {prop}]->(node)pattern, so each yielded row triggers a graph-wide scan — O(results × graph).On a moderately sized FalkorDB graph (2,852
Entity/ 18,849RELATES_TOedges) a singlesearch_memory_factsreached ~25 minutes (GRAPH.SLOWLOGlatency 1,541,600 ms) and pinned multiple cores. Thedb.idx.fulltext.queryRelationshipscall itself returns in <1 ms; the re-MATCH is the entire cost.Affected code
This re-MATCH pattern appears in two places:
graphiti_core/search/search_utils.py—edge_fulltext_search, the live path used byGraphiti.search()whendriver.search_interfaceis unset (the default).graphiti_core/driver/falkordb/operations/search_ops.py—edge_fulltext_search+edge_bfs_searchin the per-driver search operations.Fix
The fulltext / path call already returns the relationship object, so the endpoints are available directly via
startNode()/endNode()— the re-MATCH is unnecessary:edge_bfs_searchgets the same treatment, with atype(e) = 'RELATES_TO'guard sincerelationships(path)over aRELATES_TO|MENTIONSpath also yieldsMENTIONSedges that the old[e:RELATES_TO {uuid: ...}]pattern filtered out implicitly.Measurements
Verified end-to-end against a live FalkorDB instance (graphiti-core 0.28.x) — identical
search_memory_factscalls:edge_fulltext_search(search_utils.pypath)edge_fulltext_search(driversearch_ops.py)edge_bfs_search(depth 2, 289-degree origin)Correctness note (
edge_bfs_search)The old re-MATCH there was undirected —
(n:Entity)-[e:RELATES_TO {uuid: rel.uuid}]-(m:Entity)— which binds a pinned edge in both directions, returning each edge twice withsource/targetswapped (RETURN DISTINCTdoesn't collapse them).startNode(e)/endNode(e)returns each edge once in its true orientation.🤖 Generated with Claude Code