From 328deda0e527dda6f0643090828186362e2b2be2 Mon Sep 17 00:00:00 2001 From: ivscheianu Date: Wed, 26 Aug 2026 15:58:35 +0300 Subject: [PATCH 1/3] feat: report index coverage, segment count and size in SHOW INDEXES SHOW INDEXES already told an operator how many rows an index covers and how many it misses, but left the ratio to be worked out by hand -- and said nothing at all about how the index is laid out on disk. Both matter for deciding whether an index needs rebuilding, and both are already in the metadata this command fetches, so exposing them costs no extra driver work. `indexed_percent` is truncated rather than rounded so it can never overstate coverage: a table one row short of fully indexed reads 99.99, not 100. An empty table reports null rather than 100, because an index covering nothing is not up to date. This is the column that matters most in practice -- on the pinned Lance version a partially covered `zonemap` index prunes the fragments it does not cover, so a predicate on the indexed column silently returns fewer rows than the table holds while COUNT(*) over the same table stays correct. `num_segments` and `size_bytes` describe the physical layout behind one logical index. Because queries search every segment, and Lance can only compact fragments covered by an identical set of segments, a high segment count costs both query time and OPTIMIZE's ability to coalesce. Computing `size_bytes` needs the whole segment list, so the grouping now keeps every segment per name instead of dropping to the first one; when any segment predates index file size tracking the total is null, since a partial sum would understate the index rather than admit the number is unknown. --- docs/src/operations/ddl/show-indexes.md | 24 +++++- .../catalyst/plans/logical/ShowIndexes.scala | 5 +- .../datasources/v2/ShowIndexesExec.scala | 45 ++++++++++- .../spark/update/BaseShowIndexesTest.java | 75 ++++++++++++++++++- 4 files changed, 143 insertions(+), 6 deletions(-) diff --git a/docs/src/operations/ddl/show-indexes.md b/docs/src/operations/ddl/show-indexes.md index 7e47e7682..a812afaf2 100755 --- a/docs/src/operations/ddl/show-indexes.md +++ b/docs/src/operations/ddl/show-indexes.md @@ -7,7 +7,7 @@ List all indexes defined on a Lance table. ## Overview -The `SHOW INDEXES` command returns one row for each index on a Lance table. The information is retrieved using the `Dataset.describeIndices` method, and the output columns align with the attributes of `org.lance.index.IndexDescription`, excluding the per-segment metadata list. +The `SHOW INDEXES` command returns one row for each index on a Lance table. The information is retrieved using the `Dataset.describeIndices` method, and the output columns align with the attributes of `org.lance.index.IndexDescription`. Per-segment metadata is not listed individually; `num_segments` and `size_bytes` summarise it. This command is useful for inspecting existing indexes, verifying index creation, and understanding the high-level properties of each index. @@ -54,11 +54,33 @@ The `SHOW INDEXES` command returns the following columns: | `num_indexed_rows` | long | Approximate number of rows covered by the index. | | `num_unindexed_fragments` | long | Number of fragments that are not yet indexed. | | `num_unindexed_rows` | long | Approximate number of rows that are not yet covered by the index. | +| `indexed_percent` | double | Share of rows the index covers, as a percentage truncated to two decimals, so it never overstates coverage. Null for an empty table. | +| `num_segments` | long | Number of physical index segments backing this logical index. | +| `size_bytes` | long | Total size of all index files across the segments. Null if any segment predates index file size tracking. | + +## Interpreting the Output + +An `indexed_percent` below 100 means part of the table is not covered — either rows were appended +since the index was last built, or [OPTIMIZE](./optimize.md) rewrote fragments a `zonemap` or +`bloomfilter` index had covered. Rebuild with [CREATE INDEX](./create-index.md) to restore full +coverage. + +Do not treat partial coverage as merely slower. On the pinned Lance version a partially covered +`zonemap` index prunes the fragments it does not cover, so a predicate on the indexed column can +return fewer rows than the table actually holds, while a `COUNT(*)` over the same table still +reports every row. Nothing in the query plan flags the discrepancy, which is why this column is +worth checking before trusting filters over a `zonemap` index that reports less than 100. + +`num_segments` reflects how the index was built: a distributed build produces one segment per +parallel task. Queries search every segment, and Lance can only compact fragments covered by the +identical set of segments, so a high count costs both query time and `OPTIMIZE`'s ability to +coalesce. Rebuilding with [CREATE INDEX](./create-index.md) consolidates them. ## Notes - The `fields` column returns the logical column names from the Lance schema, ordered according to the index definition. - Lance-maintained system indexes, including fragment-reuse and MemWAL indexes, are excluded from the output. +- Row counts are approximate, so `indexed_percent` is a guide rather than an exact figure. ## See Also diff --git a/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ShowIndexes.scala b/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ShowIndexes.scala index 938912bec..b4f28c29a 100755 --- a/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ShowIndexes.scala +++ b/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ShowIndexes.scala @@ -47,6 +47,9 @@ object ShowIndexesOutputType { StructField("num_indexed_fragments", DataTypes.LongType, nullable = true), StructField("num_indexed_rows", DataTypes.LongType, nullable = true), StructField("num_unindexed_fragments", DataTypes.LongType, nullable = true), - StructField("num_unindexed_rows", DataTypes.LongType, nullable = true))) + StructField("num_unindexed_rows", DataTypes.LongType, nullable = true), + StructField("indexed_percent", DataTypes.DoubleType, nullable = true), + StructField("num_segments", DataTypes.LongType, nullable = true), + StructField("size_bytes", DataTypes.LongType, nullable = true))) .map(field => AttributeReference(field.name, field.dataType, field.nullable, field.metadata)()) } diff --git a/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala b/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala index 997778458..01757a164 100755 --- a/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala +++ b/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala @@ -53,15 +53,17 @@ case class ShowIndexesExec( val dataset = Utils.openDatasetBuilder(readOptions).build() try { + // Group by logical index: one row per name, with every physical segment kept so segment-level + // metadata can be aggregated. val indexes = dataset.getIndexes.asScala.toSeq .filterNot(idx => ShowIndexesExec.isSystemIndex(idx.name())) .groupBy(_.name()) .toSeq .sortBy(_._1) - .map(_._2.head) val lanceSchema = dataset.getLanceSchema() - indexes.map { idx => + indexes.map { case (_, indexSegments) => + val idx = indexSegments.head val fieldIds = idx.fields() val fieldNamesArray = if (fieldIds == null) { @@ -98,6 +100,40 @@ case class ShowIndexesExec( val numUnindexedFragments = getLong("num_unindexed_fragments") val numUnindexedRows = getLong("num_unindexed_rows") + // Share of rows the index covers, truncated to two decimals. Truncating rather than + // rounding keeps the value from ever overstating coverage: a table one row short of being + // fully indexed reads as 99.99, not as 100. Null rather than 100 for an empty table, so + // "no rows" is not reported as fully indexed either. + val indexedPercent: java.lang.Double = + if (numIndexedRows == null || numUnindexedRows == null) { + null + } else { + val total = numIndexedRows.longValue() + numUnindexedRows.longValue() + if (total <= 0L) { + null + } else { + val percent = 100.0 * numIndexedRows.longValue() / total + java.lang.Double.valueOf(math.floor(percent * 100.0) / 100.0) + } + } + + // Physical segments backing this logical index. Older cores report only `num_indices`. + val numSegments = { + val reported = getLong("num_segments") + if (reported != null) reported else getLong("num_indices") + } + + // Total across segments, or null when any segment predates index file size tracking: a + // partial sum would understate the index rather than admit it is unknown. + val sizeBytes: java.lang.Long = { + val perSegment = indexSegments.map(segment => segment.getSizeBytes) + if (perSegment.exists(!_.isPresent)) { + null + } else { + java.lang.Long.valueOf(perSegment.map(_.get.longValue()).sum) + } + } + new GenericInternalRow(Array[Any]( UTF8String.fromString(name), fieldNamesArray, @@ -105,7 +141,10 @@ case class ShowIndexesExec( numIndexedFragments, numIndexedRows, numUnindexedFragments, - numUnindexedRows)) + numUnindexedRows, + indexedPercent, + numSegments, + sizeBytes)) } } finally { dataset.close() diff --git a/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java b/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java index 0c2ff0bfd..35a97b358 100755 --- a/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java +++ b/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java @@ -112,7 +112,7 @@ public void testShowIndexes() { Dataset result = spark.sql(String.format("show indexes from %s", fullTable)); Assertions.assertEquals( - "StructType(StructField(name,StringType,true),StructField(fields,ArrayType(StringType,true),true),StructField(index_type,StringType,true),StructField(num_indexed_fragments,LongType,true),StructField(num_indexed_rows,LongType,true),StructField(num_unindexed_fragments,LongType,true),StructField(num_unindexed_rows,LongType,true))", + "StructType(StructField(name,StringType,true),StructField(fields,ArrayType(StringType,true),true),StructField(index_type,StringType,true),StructField(num_indexed_fragments,LongType,true),StructField(num_indexed_rows,LongType,true),StructField(num_unindexed_fragments,LongType,true),StructField(num_unindexed_rows,LongType,true),StructField(indexed_percent,DoubleType,true),StructField(num_segments,LongType,true),StructField(size_bytes,LongType,true))", result.schema().toString()); List rows = result.collectAsList(); @@ -138,6 +138,79 @@ public void testShowIndexes() { // num_indexed_rows should be at least 1 long numIndexedRows = row.getLong(4); Assertions.assertTrue(numIndexedRows >= 1L, "num_indexed_rows should be at least 1"); + + // a freshly created index covers every row + Assertions.assertEquals(100.0d, row.getDouble(7), 1e-9, "indexed_percent should be 100"); + + // one logical index backed by at least one physical segment + Assertions.assertTrue(row.getLong(8) >= 1L, "num_segments should be at least 1"); + + // a built index occupies storage + Assertions.assertTrue(row.getLong(9) > 0L, "size_bytes should be positive"); + } + + /** + * num_segments and size_bytes describe the whole logical index, so they have to aggregate every + * physical segment. A single-segment index cannot tell a sum from a first element, and reading + * the count from the statistics blob while summing sizes over the grouped segments means the two + * can only be trusted together if something checks they agree. + */ + @Test + public void testShowIndexesAggregatesAcrossSegments() { + spark.sql(String.format("create table %s (id int, name string) using lance", fullTable)); + for (int batch = 0; batch < 4; batch++) { + spark.sql( + String.format( + "insert into %s values (%d, 'n%d'), (%d, 'n%d')", + fullTable, batch * 2, batch * 2, batch * 2 + 1, batch * 2 + 1)); + } + spark.sql( + String.format( + "alter table %s create index test_index using btree (id) with (num_segments = 2)", + fullTable)); + + long expectedSize = 0L; + int segmentCount; + try (org.lance.Dataset dataset = + Utils.openDatasetBuilder(LanceSparkReadOptions.builder().datasetUri(tableDir).build()) + .build()) { + List segments = + dataset.getIndexes().stream() + .filter(index -> "test_index".equals(index.name())) + .collect(java.util.stream.Collectors.toList()); + segmentCount = segments.size(); + for (org.lance.index.Index segment : segments) { + expectedSize += segment.getSizeBytes().orElse(0L); + } + } + Assertions.assertEquals( + 2, segmentCount, "num_segments = 2 should have produced two physical segments"); + + Row row = spark.sql(String.format("show indexes from %s", fullTable)).collectAsList().get(0); + + Assertions.assertEquals(2L, row.getLong(8), "num_segments must count every physical segment"); + Assertions.assertEquals( + expectedSize, + row.getLong(9), + "size_bytes must sum every segment's files, not just the first"); + Assertions.assertEquals( + 100.0d, row.getDouble(7), 1e-9, "a freshly built index covers every row"); + } + + /** + * With no rows to divide, coverage is undefined rather than complete: reporting 100 would tell an + * operator polling for staleness that an index covering nothing is up to date. + */ + @Test + public void testShowIndexesReportsNullPercentForEmptyTable() { + spark.sql(String.format("create table %s (id int, text string) using lance;", fullTable)); + spark.sql(String.format("alter table %s create index test_index using btree (id)", fullTable)); + + Row row = spark.sql(String.format("show indexes from %s", fullTable)).collectAsList().get(0); + + Assertions.assertTrue(row.isNullAt(7), "indexed_percent should be null for an empty table"); + Assertions.assertEquals(0L, row.getLong(4), "no rows can be indexed"); + Assertions.assertEquals(0L, row.getLong(6), "no rows can be unindexed"); } @Test From ae561fa244f2139323de0c7c1f6076a1752c005d Mon Sep 17 00:00:00 2001 From: ivscheianu Date: Wed, 2 Sep 2026 21:28:50 +0300 Subject: [PATCH 2/3] docs: remove stale zonemap warning and trim comments The zonemap partial-coverage bug was fixed by #781, so the documentation should not warn about it. Also trim inline comments and test Javadocs to match the code, not the PR history. --- docs/src/operations/ddl/show-indexes.md | 11 ++--------- .../execution/datasources/v2/ShowIndexesExec.scala | 5 +---- .../org/lance/spark/update/BaseShowIndexesTest.java | 10 ---------- 3 files changed, 3 insertions(+), 23 deletions(-) diff --git a/docs/src/operations/ddl/show-indexes.md b/docs/src/operations/ddl/show-indexes.md index a812afaf2..6fa995a17 100755 --- a/docs/src/operations/ddl/show-indexes.md +++ b/docs/src/operations/ddl/show-indexes.md @@ -61,15 +61,8 @@ The `SHOW INDEXES` command returns the following columns: ## Interpreting the Output An `indexed_percent` below 100 means part of the table is not covered — either rows were appended -since the index was last built, or [OPTIMIZE](./optimize.md) rewrote fragments a `zonemap` or -`bloomfilter` index had covered. Rebuild with [CREATE INDEX](./create-index.md) to restore full -coverage. - -Do not treat partial coverage as merely slower. On the pinned Lance version a partially covered -`zonemap` index prunes the fragments it does not cover, so a predicate on the indexed column can -return fewer rows than the table actually holds, while a `COUNT(*)` over the same table still -reports every row. Nothing in the query plan flags the discrepancy, which is why this column is -worth checking before trusting filters over a `zonemap` index that reports less than 100. +since the index was last built, or [OPTIMIZE](./optimize.md) rewrote fragments an index had +covered. Rebuild with [CREATE INDEX](./create-index.md) to restore full coverage. `num_segments` reflects how the index was built: a distributed build produces one segment per parallel task. Queries search every segment, and Lance can only compact fragments covered by the diff --git a/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala b/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala index 01757a164..84d2a7779 100755 --- a/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala +++ b/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala @@ -100,10 +100,7 @@ case class ShowIndexesExec( val numUnindexedFragments = getLong("num_unindexed_fragments") val numUnindexedRows = getLong("num_unindexed_rows") - // Share of rows the index covers, truncated to two decimals. Truncating rather than - // rounding keeps the value from ever overstating coverage: a table one row short of being - // fully indexed reads as 99.99, not as 100. Null rather than 100 for an empty table, so - // "no rows" is not reported as fully indexed either. + // Truncated (not rounded) so it never overstates coverage. Null for empty tables. val indexedPercent: java.lang.Double = if (numIndexedRows == null || numUnindexedRows == null) { null diff --git a/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java b/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java index 35a97b358..48af5bc09 100755 --- a/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java +++ b/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java @@ -149,12 +149,6 @@ public void testShowIndexes() { Assertions.assertTrue(row.getLong(9) > 0L, "size_bytes should be positive"); } - /** - * num_segments and size_bytes describe the whole logical index, so they have to aggregate every - * physical segment. A single-segment index cannot tell a sum from a first element, and reading - * the count from the statistics blob while summing sizes over the grouped segments means the two - * can only be trusted together if something checks they agree. - */ @Test public void testShowIndexesAggregatesAcrossSegments() { spark.sql(String.format("create table %s (id int, name string) using lance", fullTable)); @@ -197,10 +191,6 @@ public void testShowIndexesAggregatesAcrossSegments() { 100.0d, row.getDouble(7), 1e-9, "a freshly built index covers every row"); } - /** - * With no rows to divide, coverage is undefined rather than complete: reporting 100 would tell an - * operator polling for staleness that an index covering nothing is up to date. - */ @Test public void testShowIndexesReportsNullPercentForEmptyTable() { spark.sql(String.format("create table %s (id int, text string) using lance;", fullTable)); From 04c9ce1fb5a7238acce69d3168ef92c8e00d9ac4 Mon Sep 17 00:00:00 2001 From: ivscheianu Date: Wed, 2 Sep 2026 22:00:15 +0300 Subject: [PATCH 3/3] style: remove redundant qualifications and explicit boxing --- .../sql/execution/datasources/v2/ShowIndexesExec.scala | 4 ++-- .../java/org/lance/spark/update/BaseShowIndexesTest.java | 7 ++++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala b/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala index 84d2a7779..da42f0aae 100755 --- a/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala +++ b/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowIndexesExec.scala @@ -110,7 +110,7 @@ case class ShowIndexesExec( null } else { val percent = 100.0 * numIndexedRows.longValue() / total - java.lang.Double.valueOf(math.floor(percent * 100.0) / 100.0) + math.floor(percent * 100.0) / 100.0 } } @@ -127,7 +127,7 @@ case class ShowIndexesExec( if (perSegment.exists(!_.isPresent)) { null } else { - java.lang.Long.valueOf(perSegment.map(_.get.longValue()).sum) + perSegment.map(_.get.longValue()).sum } } diff --git a/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java b/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java index 48af5bc09..fdb631cba 100755 --- a/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java +++ b/lance-spark-base_2.12/src/test/java/org/lance/spark/update/BaseShowIndexesTest.java @@ -13,6 +13,7 @@ */ package org.lance.spark.update; +import org.lance.index.Index; import org.lance.index.IndexOptions; import org.lance.index.IndexParams; import org.lance.index.IndexType; @@ -168,12 +169,12 @@ public void testShowIndexesAggregatesAcrossSegments() { try (org.lance.Dataset dataset = Utils.openDatasetBuilder(LanceSparkReadOptions.builder().datasetUri(tableDir).build()) .build()) { - List segments = + List segments = dataset.getIndexes().stream() .filter(index -> "test_index".equals(index.name())) - .collect(java.util.stream.Collectors.toList()); + .collect(Collectors.toList()); segmentCount = segments.size(); - for (org.lance.index.Index segment : segments) { + for (Index segment : segments) { expectedSize += segment.getSizeBytes().orElse(0L); } }