[FEATURE] Native top-level Variant projection from ordinary Parquet

[peterxcli]

What problem does this solve?

#5407 implements the ordinary-Parquet phase of #4295, but it now combines several independently reviewable concerns: shared Arrow FFI schema export, Variant logical-type transport, native whole-value normalization, Spark-compatible Parquet physical semantics, and scan admission/schema-evolution defaults.

This issue tracks re-landing that work as five atomic pull requests under the broader Variant roadmap in #5438, following the maintainer request in #5407 (comment).

Scope

Support direct, top-level, whole-value VariantType projection from ordinary Parquet:

SELECT v FROM parquet_table;
SELECT id, v, tail FROM parquet_table;

The completed series preserves Variant identity through:

Spark VariantType
  -> Comet protobuf
  -> marked Arrow Field
  -> ordinary native Parquet scan
  -> whole-value normalization
  -> Arrow C Data Interface
  -> CometStructVector
  -> ColumnVector.getVariant

This tracker does not include Iceberg, nested Variant, Variant expressions, subfield pruning, predicate pushdown, writes, columnar-to-row, shuffle/spill, or Python transport.

Ordered subtasks

• [FEATURE] Export complete Arrow Fields through the native FFI boundary #5547 — Export complete Arrow Fields through the native FFI boundary (fix: preserve Arrow Field metadata across C Data exports #5552)
• [FEATURE] Carry Spark VariantType identity through Comet schema serialization #5548 — Carry Spark VariantType identity through Comet schema serialization
• [FEATURE] Normalize marked Variant arrays at the native Parquet boundary #5549 — Normalize marked Variant arrays at the native Parquet boundary
• [FEATURE] Match Spark physical Parquet semantics for projected Variant columns #5550 — Match Spark physical Parquet semantics for projected Variant columns
• [FEATURE] Enable direct top-level Variant projection in ordinary native Parquet scans #5551 — Enable direct top-level Variant scan admission, defaults, and fallbacks

Each pull request must include focused tests and retain explicit Spark fallback until the complete scan boundary is ready.

Definition of done

• Direct top-level Variant projection remains a Comet native Parquet scan and matches Spark.
• Fields before and after Variant retain their order.
• Objects, arrays, scalars, Variant JSON null, SQL NULL, and nullable parents round-trip.
• Canonical unshredded and shredded inputs reconstruct whole values.
• Spark receives logical VariantType with exactly [value, metadata] children.
• Unread Variant pruning from feat: Support native scans with unprojected Spark 4 VARIANT columns #5377 remains native.
• Unsupported expressions, nested Variant, pushed VariantStruct, native writes, C2R, shuffle/spill, Python operators, and Iceberg retain explicit fallback.
• Spark 3.x compilation and behavior remain unchanged.

Related work

• [FEATURE] Shredded Parquet Reader/Writer support for Variant type #3983 — shredded writing, pruning, and predicate pushdown
• Remove Variant UTF-16 output rewriting #5474 — removal of legacy Spark UTF-16 output rewriting
• Track and remove native Variant compatibility workarounds after upstream fixes #5477 — removal or reassessment of upstream compatibility paths
• Support Spark-compatible Unicode case-insensitive Parquet field matching #5495 — Spark-compatible Unicode Parquet field matching
• Support Spark's pushed one-field VariantStruct (whole-value pushVariantIntoScan rewrite) in native Parquet scans #5519 — pushed one-field VariantStruct

[peterxcli]

take