fix!: preserve unindexed fragments during zonemap pruning

[ddupg]

Summary

• preserve live fragments that are not covered by Zonemap stats during driver-side pruning
• ignore retired fragment stats and require full live-fragment coverage before advertising storage partitioning
• weight post-pruning row and size estimates by the surviving fragment row counts
• add regression coverage for append-after-index filtering and partial-coverage SPJ fallback

Problem

Zonemap stats only describe indexed fragments. After an append, treating the returned stats as complete could drop the new fragment before Lance scans it. The same assumption could advertise KeyGroupedPartitioning without a trustworthy key for every live fragment.

Testing

• ./mvnw test -pl lance-spark-3.5_2.13 -am -Dtest=LanceStatisticsTest,ZonemapFragmentPrunerTest,SparkLanceShardingUtilsTest,LanceScanTest -Dsurefire.failIfNoSpecifiedTests=false
• ./mvnw test -pl lance-spark-3.5_2.12 -am -Dtest=BucketSpjTest -Dsurefire.failIfNoSpecifiedTests=false
• final LanceStatisticsTest run: 16 tests passed
• Spotless, Checkstyle, Python integration-test syntax check, and git diff --check passed

[lance-gatekeeper]

✅ Gate recommendation: approve.

The rebased implementation preserves the prior result. Resolving the LanceRef into the scan plan and passing that resolved reference to workers keeps live-fragment discovery, zonemap statistics, split planning, and reads on the same dataset snapshot.