Skip to content

Remove vendored PushDownLeafProjections workaround after DataFusion upgrade #1232

Description

@paleolimbot

Problem

SedonaDB currently depends on DataFusion 54.1.0, which contains a logical optimizer correctness bug in PushDownLeafProjections when nested-field access is applied to a struct produced by Unnest.

For a plan shaped like:

Projection: get_field(dump, "geom")
  Unnest: dump
    Projection: ST_Dump(geometry) AS dump

the optimizer can push get_field(dump, "geom") below Unnest. At that point dump is still List<Struct<...>>, rather than the post-unnest Struct<...>, and optimization fails with:

Optimizer rule 'push_down_leaf_projections' failed
caused by
Execution error: Cannot access field at argument 1: type List(Struct(...))
is not Struct, Map, or Null

This is a plan-correctness bug, not a security issue.

Upstream status

The bug was reported and fixed upstream:

The reduced reproducer fails with DataFusion 54.1.0 and succeeds on DataFusion main at e1942b144.

Temporary SedonaDB workaround

Until SedonaDB upgrades to a DataFusion release containing apache/datafusion#22620, sedona-query-planner replaces DataFusion's optimizer rule named push_down_leaf_projections with a compatibility rule that prevents leaf projection pushdown across Unnest.

Regression coverage includes both a DataFusion-only List<Struct> plan and SedonaDB's ST_Dump → unnest → geom workflow.

Action item

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions