Agentic insights: intent + cross-pipeline relationships in the discover inventory

[matthewmoorcroft]

⚠️ Draft — initial port. This change is rebuilt onto the unified base as-is. It predates the engine → sources/adf module restructure, so it still needs adaptation to the new layout (and a green test run) before it is mergeable.

📦 Merge order (3-PR set): This PR is stacked on the discover-lineage PR (deterministic lineage) — its diff shows only the insights delta, and it must merge after the discover-lineage PR (the code imports build_lineage and validates edge_identity against inventory.lineage, both introduced by the discover-lineage PR). Base is feat/discover-lineage deliberately; GitHub auto-retargets this to main once the discover-lineage PR merges. Independent of the Switch-parsing PR (Switch parser fix), which can merge in any order.

---

What

Adds an agentic insights layer to the discover-phase inventory.json: an LLM authors an insights object (an overview, a sparse pipeline_insights[], and pipeline_relationships[]) that is validated against the deterministic inventory and merged in. This recovers the intent and cross-pipeline coupling that per-activity deterministic analysis structurally cannot see.

Stacked on feat/discover-lineage (PR the discover-lineage PR) — review/merge that first.

Key pieces

• enrich phase — validate-then-append two-pass write; MCP enrich command + adapter subcommand; materialize_json runner support.
• Two-tier lineage_edge model:
  • Annotation (control / data) — echoes an edge the deterministic layer already found.
  • Inferred — a real coupling the deterministic layer could not represent (data written in notebook code, sibling dependsOn ordering, shared control/config assets), carrying mandatory evidence + confidence. This is what captures data flow on notebook-centric factories where data_edges is empty.
• Sound annotation resolution — control/data annotations resolve on the full (from, to, edge_identity) triple, not a global identity set, so a mis-paired annotation cannot validate (ADF shares one activity name across many caller→callee edges).
• Research-backed skill Step 5 — criteria-based sparse selection, principle-first inferred/annotation boundary, and the correct flat ARM file shape.
• Optional has_insights coverage column.

Verification

• Full unit suite green; insights validator suite 41 tests incl. triple-resolution regressions (shared-callee, shared match_key, reversed direction, right-pair/wrong-identity).
• Two independent clean-room runs on a real 327-pipeline factory: both enriched with 0 validation violations, both used the inferred tier for real data couplings the deterministic layer missed, and both independently surfaced the annotation-identity non-uniqueness that the triple-resolution commit fixes.
• Adversarial review of the triple-resolution fix confirmed the from→caller/producer, to→callee/consumer mapping direction and that the regression tests fail against the pre-fix code.

Not covered / notes

• 6 pre-existing convert/SetVariable golden-output integration failures are present on the base branch too (verified identical failures at the parent commit) — unrelated to this change.

This pull request and its description were written by Isaac.