Skip to content

Parse Switch case/default activities (fixes undercounted inventory & missing lineage edges) - #38

Open
matthewmoorcroft wants to merge 2 commits into
unify/reconcile-internalfrom
pr/switch-case-activities
Open

Parse Switch case/default activities (fixes undercounted inventory & missing lineage edges)#38
matthewmoorcroft wants to merge 2 commits into
unify/reconcile-internalfrom
pr/switch-case-activities

Conversation

@matthewmoorcroft

Copy link
Copy Markdown
Member

⚠️ Draft — initial port. This change is rebuilt onto the unified base as-is. It predates the engine → sources/adf module restructure, so it still needs adaptation to the new layout (and a green test run) before it is mergeable.

📦 Merge order (3-PR set): This PR is independent — it branches off main and depends on nothing else; merge it any time. The other two (the discover-lineage PR lineage → the discover-insights PR insights) are a stack that benefits from this fix but does not require it. Files overlap with the discover-lineage PR/the discover-insights PR only in disjoint regions, so at most a trivial rebase for whichever merges second.


Fixes #31

Problem

parse_activity in src/flowx/parser/adf_loader.py descends into ifTrueActivities/ifFalseActivities (IfCondition) and activities (ForEach/Until), but never cases[].activities or defaultActivities on a Switch. So a Switch's nested activities never entered the AdfActivity AST, and every AST consumer inherited the blindness:

  • Inventory (_classify_activities) undercounted activity_count / coverage_pct.
  • Lineage (_walk_activities) dropped every ExecutePipeline and data edge nested inside a Switch case.

Why this is a bug, not a design choice

Verified via git archaeology and the repo's own docs:

  • Switch case-parsing never existed in the parser across its entire history (git log --all -S 'defaultActivities' / -S '"cases"' return only this commit) — nothing was removed "for a reason."
  • The Switch translator (translator/activity_translators/switch.py) has read cases[].activities + defaultActivities since the initial release — one half of the tool always understood the shape; the parser half didn't.
  • The repo's own docs assert the behavior this restores: flowx-discover/SKILL.md documents the activities count as "including nested ForEach/If/Switch children", and flowx-convert/SKILL.md documents nested gaps inside a Switch as reported. The code contradicted its own spec.
  • The commit that consciously improved control-flow parsing added "Switch" to the container type-sets but never added case-body parsing — an oversight in that change, not an exclusion. No TODO/FIXME anywhere flags Switch as intentionally unsupported.

Fix

  • Add switch_cases (case value → activities) and switch_default_activities to AdfActivity, plus a switch_child_activities() helper for a flat view.
  • Parse both in parse_activity.
  • Descend into them in _classify_activities (inventory) and _walk_activities (lineage).

The translate/bundle path is untouched — it reads raw typeProperties, not these new fields.

Verification

  • Verified on the real app0001 export (327 pipelines): activity_count 1,804 → 2,159 (+355 across 38 pipelines); ExecutePipeline calls reachable by the lineage walker 599 → 808 (~209 edges recovered, ~26% of the true control graph — including orchestrator_main's entire fan-out, previously a near-leaf).
  • Full suite: 852 passing (+4 new regression tests covering parse, the helper, inventory count, and lineage reachability); ruff + mypy clean.
  • 6 pre-existing convert/SetVariable golden-output failures are present on main unchanged (verified identical at the base commit) — unrelated to this change.

Note on stacked branches

This targets main. The in-flight feat/discover-lineage (PR the discover-lineage PR) and feat/discover-insights (PR the discover-insights PR) both sit on the buggy parser, so their lineage/inventory numbers improve once this lands and they rebase.

This pull request and its description were written by Isaac.

@matthewmoorcroft
matthewmoorcroft marked this pull request as ready for review September 1, 2026 12:16
Rewrite internal package-proxy URLs (pypi-proxy.dev.databricks.com) to
pypi.org / files.pythonhosted.org so public CI resolves deps. Same pinned
versions and hashes; matches main.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant