Skip to content

Agentic insights: intent + cross-pipeline relationships in the discover inventory - #42

Open
matthewmoorcroft wants to merge 8 commits into
pr/discover-lineagefrom
pr/discover-insights
Open

Agentic insights: intent + cross-pipeline relationships in the discover inventory#42
matthewmoorcroft wants to merge 8 commits into
pr/discover-lineagefrom
pr/discover-insights

Conversation

@matthewmoorcroft

@matthewmoorcroft matthewmoorcroft commented Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Agentic insights: intent + cross-pipeline relationships in the discover inventory

Closes #25

What this adds

An optional agentic insights layer on top of the deterministic discover inventory. After discover writes metadata/inventory.json (pipelines, summary, deterministic lineage), an agent authors an insights object capturing judgment the deterministic pass can't derive, and flowx enrich validates it and merges it back under a single insights key.

  • Per-pipeline insight — the pipeline's intent, a mapped Databricks pattern, a ranked list of recommended_patterns (≤4, each flagged as a genuine simplification or a like-for-like port), conversion notes, and risk-if-ignored.
  • Cross-pipeline relationships — how pipelines couple, each anchored to a typed lineage_edge.
  • System recommendation — one factory-wide architectural decision (headline + cascade + gating driver) that the per-pipeline patterns are chosen under.

New source-neutral models in models/insights.py (Insights, PipelineInsight, PipelineRelationship, LineageEdgeRef, RecommendedPattern, SystemRecommendation), a parser/pipeline_insights.py engine, an enrich command on both the adapter CLI and the MCP server, and a has_insights column in the coverage results table.

How it works

The tool contains no LLM. The agent authors the JSON; pipeline_insights.py only validates and merges, so the deterministic inventory stays trustworthy and every insight is accountable. enrich (CLI enrich --insights-path, or MCP enrich with an inline dict) runs enrich_inventory, which:

  1. Validates the authored JSON against the real inventory (validate_insights):
    • Every pipeline and every relationship endpoint must be a real pipeline name in the inventory.
    • Each relationship's lineage_edge is validated in two tiers:
      • control / data — an annotation of a deterministic edge: edge_identity must resolve to a real ControlEdge.activity_name (control) or DataEdge.match_key (data) already in the inventory's lineage; evidence/confidence must be omitted.
      • inferred — a coupling the deterministic layer never found (e.g. data flow buried in notebook code): nothing to resolve against, so it must carry a non-empty evidence string and a confidence of high/medium/low.
    • recommended_patterns is a ranked list of ≤4, each with a simplification_pattern flag; simplifying patterns rank first.
  2. Merges only on clean validation (merge_into_inventory): appends the single insights key and re-serialises the rest of inventory.json byte-identically (non-destructive enrichment).

Schema added to inventory.json

A single top-level insights key:

"insights": {
  "overview": "string | null",                     // factory-wide summary
  "system_recommendation": {                        // optional
    "headline": "string",                           // required
    "recommended_patterns": [ /* RecommendedPattern[], 1-4, best-first */ ],
    "cascade": ["string", ...],                     // what picking [0] collapses; [] if none
    "decision_driver": "string | null"              // the gating question; null if none
  },
  "pipeline_insights": [
    {
      "pipeline": "string",                         // required; must be a real pipeline name
      "pattern_name": "string | null",
      "intent": "string | null",
      "databricks_pattern": "string | null",
      "recommended_patterns": [ /* RecommendedPattern[], 1-4 */ ],
      "conversion_notes": ["string", ...],
      "risk_if_ignored": "string | null"
    }
  ],
  "pipeline_relationships": [
    {
      "from_pipeline": "string",                    // required; real pipeline name
      "to_pipeline": "string",                      // required; real pipeline name
      "lineage_edge": {
        "edge_type": "control | data | inferred",   // required
        "edge_identity": "string",                  // control: ControlEdge.activity_name,
                                                     // data: DataEdge.match_key, inferred: descriptor
        "evidence": "string | null",                // required iff inferred; else omitted
        "confidence": "high | medium | low | null"  // required iff inferred; else omitted
      },
      "relationship_summary": "string | null",
      "databricks_pattern": "string | null",
      "risk_if_ignored": "string | null"
    }
  ]
}

RecommendedPattern (used by both system_recommendation and each pipeline insight):

{ "pattern": "string", "fit": "string", "simplification_pattern": true }

Relationship to the lineage epic (#24)

The cross-pipeline relationships here are the discover-side signal that #24 (ordered cross-pipeline deploy/run in the package phase) builds on: annotation edges resolve against #23's deterministic lineage.control_edges / data_edges, while inferred edges surface couplings the deterministic layer cannot yet see.

Sources (ADF & Airflow)

Insights are source-agnostic: they attach to the discover inventory.json (which both sources/adf and sources/airflow emit), not to any ADF internals. The engine (pipeline_insights.py), the enrich command, and the models are source-neutral; only the discover skill's insight step carries source specifics, and Step 5 now branches those by --source — the source deep-dive (ADF ARM *.arm.json vs Airflow DAG source) and the recommended-pattern vocabulary (ADF-constructs table + Airflow-operators table).

Caveat: control/data annotation edges resolve against the inventory's deterministic lineage, which today only sources/adf emits. On an Airflow inventory those edges have nothing to resolve against, so only per-pipeline insights and inferred edges validate — graceful degradation. Emitting deterministic lineage from sources/airflow is a tracked follow-up.

Testing

Full unit suite: 1306 passed with the mcp extra; 1224 passed / 3 skipped without it. The MCP-dependent test_pipeline_insights.py is guarded with pytest.importorskip("mcp") (matching the existing test_mcp_* convention), since CI does not install the optional mcp extra. make fmt + mypy clean.

This pull request and its description were written by Isaac.

@matthewmoorcroft
matthewmoorcroft marked this pull request as ready for review September 1, 2026 12:17
matthewmoorcroft and others added 6 commits September 1, 2026 13:43
@matthewmoorcroft Isaac
Point uv.lock at public PyPI
826a19c 
Rewrite internal package-proxy URLs (pypi-proxy.dev.databricks.com) to
pypi.org / files.pythonhosted.org so public CI resolves deps. Same pinned
versions and hashes; matches main.

Co-authored-by: Isaac <no-reply@databricks.com>
@matthewmoorcroft Isaac
Merge pr/discover-lineage into discover-insights; adapt insights to t…
63c4414 
…he unified base

Reconciles the agentic-insights work with its base (#36) via merge, no history
rewrite. The insights branch was an initial port predating the engine ->
sources/adf restructure and #36's lineage + reporting schema-evolution work, so
the merge resolves to:

- #36's canonical lineage / dataset-resolver files (the insights branch's
  near-identical duplicates are dropped in favour of the base), keeping only the
  insights payload: pipeline_insights, the insight AST models, and the
  MCP / adapter / reporting wiring.
- execute_pipeline.py + test_pipeline_insights.py: imports repointed to the new
  layout (flowx.sources.adf.translators.resolve, flowx.sources.adf.loader).
- test_pipeline_insights.py guarded with pytest.importorskip("mcp") so CI -- which
  does not install the optional mcp extra -- skips it, matching
  test_mcp_migrate / test_mcp_source_routing.
- reporting/results.py: full _STRING_METRICS / _FLOAT_METRICS classification
  restored (only has_insights is new), fixing the int() crash on string / float
  coverage columns.
- test_reporting_results.py: expects the base's CREATE -> SHOW COLUMNS -> INSERT
  sequence.

Full unit suite: 1306 passed with the mcp extra; 1224 passed / 3 skipped without
it. make fmt + mypy clean.

Co-authored-by: Isaac <no-reply@databricks.com>
@matthewmoorcroft Isaac
Make agentic insights source-agnostic (ADF + Airflow)
9780981 
Airflow is now a first-class deterministic source, so the insights layer should
apply to both. The engine (parser/pipeline_insights.py) and the enrich command
were already source-neutral (they operate on the inventory dict); this closes the
two remaining ADF couplings:

- Move the insight models (Insights, PipelineInsight, PipelineRelationship,
  LineageEdgeRef, RecommendedPattern, SystemRecommendation) out of the ADF AST
  module into a source-neutral models/insights.py -- they are used only by the
  tests, the runtime path is dict-based. Update the test import.
- flowx-discover SKILL.md Step 5: keep the neutral authoring core (schema,
  sparse/ranked philosophy, edge-accountability model, Databricks target
  vocabulary) shared, and branch the two genuinely source-specific pieces by
  --source: the source deep-dive (ADF ARM *.arm.json vs Airflow DAG source) and
  the pattern vocabulary (ADF constructs table + new Airflow operators table).
  Neutralise incidental ADF wording in the edge-authoring rules.

Out of scope: emitting deterministic lineage from sources/airflow (tracked
separately). Until then an Airflow inventory has no lineage block, so only
per-pipeline insights and inferred edges validate there -- control/data
annotation edges degrade gracefully, no change needed here.

Full unit suite: 1306 passed with the mcp extra; 1224 passed / 3 skipped without
it. make fmt + mypy clean.

Co-authored-by: Isaac <no-reply@databricks.com>
@matthewmoorcroft Isaac
Move per-source insight guidance into the discover source guides
f71651b 
Follow the discover skill's own hub-and-spoke convention (SKILL.md: "the shared
mechanics live here"; Step 2: "read the matching sources/<source>.md and follow
it"). The insights authoring step had inlined ADF/Airflow specifics into the
shared SKILL.md.

- SKILL.md Step 5 keeps the source-neutral core (schema, analysis method, pattern
  framework, edge model, enrich) and now points to the source guide for the source
  deep-dive and the construct->Databricks pattern vocabulary. Marked the step
  explicitly source-neutral ("runs for every source").
- sources/adf.md and sources/airflow.md each gain an "Insights -- deep-dive &
  pattern vocabulary" section (ARM *.arm.json / DAG-source deep-dive + the
  source-construct pattern table) and a pointer back to the shared authoring step.

This also wires insights for Airflow: an Airflow run follows sources/airflow.md
(per Step 2 routing), which now carries the source deep-dive + pattern table and
routes to the shared authoring+enrich step. Previously that step lived only inside
SKILL.md's ADF-specific ## Workflow, so an Airflow run never reached it.

Docs only; unit suite unchanged (1224 passed / 3 skipped without the mcp extra).

Co-authored-by: Isaac <no-reply@databricks.com>
@matthewmoorcroft Isaac
Decompose flowx-discover SKILL.md back to main's hub structure
ae6fd45 
The branch had re-introduced a monolithic ADF-specific `## Workflow` into
skills/flowx-discover/SKILL.md (518 lines), which main had already decomposed into
a source-neutral hub (80 lines) + per-source guides. Merging as-is would have
reverted main's decomposition of this one skill; every other flowx skill already
matched main.

Rebuild SKILL.md on main's clean hub (Identify source -> Follow source guide ->
How to run -> Output artifacts (shared) -> Reference) and add ONE shared section,
"Author and merge agentic insights (all sources)", holding the source-neutral
insight core (schema, analysis method, pattern framework, edge model, enrich). An
Airflow run now loads zero ADF walkthrough into context.

Preserve the two pieces of new value that lived only in the monolith by moving
them into the source guides:
- sources/adf.md: the inventory `lineage` block + its explanation, and the
  insights read-back in the summary step.
- sources/airflow.md: the insights read-back, plus a note that Airflow inventories
  carry no `lineage` block yet (so cross-DAG relationships use `inferred` edges).

Docs only; unit suite unchanged (1224 passed / 3 skipped without the mcp extra).

Co-authored-by: Isaac <no-reply@databricks.com>
@matthewmoorcroft Isaac
Keep #42 focused on agentic insights: drop deterministic-lineage docs
d28275a 
Deterministic lineage (the inventory `lineage` block) is owned by the separate
lineage PR that #42 builds on top of; it is not #42's to document. The previous
commit had pulled the `lineage` block + its explanation into sources/adf.md and a
lineage-state note into sources/airflow.md -- scope creep into the lineage PR.

Revert both: sources/adf.md's inventory example returns to main's (no `lineage`
block) and the airflow.md discovery step drops the lineage note. The discover docs
on this branch are now main + insights-only additions.

The shared insights authoring section still *consumes* lineage (the annotation vs
inferred edge model resolves against `lineage.control_edges` / `data_edges`) -- a
dependency on the lineage PR, not documentation of it.

Docs only; unit suite unchanged (1224 passed / 3 skipped without the mcp extra).

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@matthewmoorcroft