fix(compaction): avoid stranded remainder fragments

[lance-gatefixer]

Summary

• derive target-scale output counts from surviving rows and balance remainders across outputs
• keep ordinary planner-sized compaction tasks in one fragment without allowing oversized tasks to collapse into one file
• cover both the reported small-fragment layout and an oversized deletion rewrite next to a small fragment

Root cause

The planner already groups adjacent small fragments into tasks with approximately target_rows_per_fragment rows. Execution applied that target again as a hard max_rows_per_file, so each task above the target produced a full fragment followed by a small remainder that could become isolated. Removing the cap solely based on source-fragment count then let an indivisible oversized source collapse into one arbitrarily large output.

The revised executor computes the output count from logical surviving rows and balances the tail across that count. Planner-sized tasks still produce one output, while genuinely oversized tasks remain target-scale.

Validation

• cargo test -p lance --lib dataset::optimize::tests (123 passed)
• cargo fmt --all
• cargo clippy --all --tests --benches -- -D warnings
• make install and make build from python/
• uv run pytest python/tests/test_scalar_index.py::test_zonemap_index_remapping
• uv run make lint

Fixes #8506

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[lance-gatefixer]

Addressed in 5a9234d: planned row-group sizes are now generated and consumed lazily by the shared chunker, preserving the streaming memory bound. The chunker regression verifies that no sizes are consumed at construction and that sizes advance only as chunks are read.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The merged revision retains balanced Stable and Legacy compaction boundaries: planner-sized tasks avoid stranded remainders, while byte-limited and oversized rewrites replan remaining rows within the target-scale count. Cleanup and row-ID/index remapping behavior remain preserved.