RFC: C-level primitives for distributed index builds (fragment-scoped uncommitted builds, segment metadata, progress, cancellation)

[u70b3]

RFC: C-level primitives for distributed index builds

Context. I'm a contributor on Apache Doris, working on apache/doris#66497 — a Lance index lifecycle where C++ backend workers drive lance-c and a Java coordinator drives the Lance Java SDK. The downstream design builds its post-4.2 distributed track on these six extensions; the 4.2 release itself is scoped to the pinned v0.1.2 one-shot APIs, so nothing in this RFC gates a downstream release. I'd like to upstream these capabilities here rather than carry a fork. All claims below are pinned to lance-c @ v0.1.6 (98e4d54) and lance @ 9.1.0-beta.3; I intend to implement them myself as upstream PRs, in whatever shape maintainers prefer.

1. Summary

lance-c today can create, drop, list, and — since v0.1.3 — enumerate and consume index segments for distributed search. It cannot participate in distributed build: there is no way to build an index segment over a fragment subset without committing, no way to inject a caller-trained shared model, no way to move segment metadata between processes, no progress or cancellation channel, and no per-segment coverage/stats exposure.

The key finding: the Rust core already implements nearly all of these primitives — what is missing is almost entirely FFI exposure. I propose six extensions as a four-PR sequence plus one separate track:

#	Extension	PR	Core primitive status (pinned)	Work required
E1	Fragment-scoped uncommitted segment build, incl. shared-model injection	1 (foundational)	Exists — CreateIndexBuilder.fragments/index_uuid/train/execute_uncommitted	FFI only
E2	Segment metadata serialization (in/out)	1 (foundational)	Exists — IndexMetadata ↔ pb::IndexMetadata prost round-trip	FFI only
E5	Commit-existing-segments C API	2	Exists — Dataset::commit_existing_index_segments	FFI only
E3	Build progress reporting	3	Exists — IndexBuildProgress trait	FFI bridge
E6	Per-segment coverage/stats in index listing	4	Missing from listing — data exists in IndexMetadata, list JSON is hard-coded	FFI (format extension)
E4	Cooperative cancellation	separate track	Missing — no cancel hook in the create path	Core + FFI

The PRs are separable by design: PR 1 unblocks a full distributed build loop (worker builds segment → ships metadata bytes → coordinator commits via any SDK that can commit IndexMetadata); PR 2 gives C-only embedders the commit half; PRs 3–4 are production-quality and observability items. E4 runs as a separate track because its core token work lands in lance first (§9).

2. Motivation

Lance's own distributed indexing guide describes the shape: a coordinator assigns disjoint fragment subsets to stateless workers; each worker builds an uncommitted segment; the coordinator commits all segments under one logical index in a single dataset version. The Java SDK implements this loop (IndexOptions.withFragmentIds/withIndexUUID, VectorTrainer, Dataset.commitExistingIndexSegments); I verified it end-to-end with a PoC on lance 9.1.0-beta.3 (fragment-scoped uncommitted builds over disjoint subsets → single coordinator commit → dataset version +1 → k-NN consumes the multi-segment logical index). Happy to publish the PoC harness if useful.

lance-c has the consumption half: lance_dataset_index_segment_count / lance_dataset_index_segments enumerate segment UUIDs and lance_scanner_set_index_segments pins scan-time segment selection (lance.h:976,994,1045). The build half is absent: the only creation surface is the one-shot, internally-committing lance_dataset_create_vector_index / lance_dataset_create_scalar_index (lance.h:932,945; both call DatasetIndexExt::create_index, which commits internally — lance-c src/index.rs:607-615 / :120-128).

3. Verified status quo

3.1 What lance-c v0.1.6 exposes today (index lifecycle)

• One-shot committed creation: lance_dataset_create_vector_index (lance.h:932), lance_dataset_create_scalar_index (lance.h:945) — vector columns restricted to FixedSizeList<float32|float16|uint8|int8> (lance.h:927).
• lance_dataset_drop_index (lance.h:955), lance_dataset_index_count (lance.h:958).
• Listing: lance_dataset_index_list_json (lance.h:965) — per-entry JSON hard-coded to {name, uuid, columns, type, dataset_version} (lance-c src/index.rs:361-368); no fragment coverage, no row counts, one entry per physical segment.
• Segment enumeration for distributed search: lance_dataset_index_segment_count / lance_dataset_index_segments (lance.h:976,994; impl src/index.rs:157-295), consumed by lance_scanner_set_index_segments (lance.h:1045).

3.2 What lance core 9.1.0-beta.3 already has (not FFI-exposed)

All in rust/lance/src/index/create.rs, pub struct CreateIndexBuilder (:50):

• fragments(Vec<u32>) (:104) — build over a fragment subset. Honored for scalar indexes too — including an explicit IndexType::Bitmap && fragments.is_some() path (:270) and fragment plumbing at :282-350.
• index_uuid(Uuid) (:109) — caller-assigned segment UUID (coordinator pre-assignment, so the expected UUID set is known before dispatch).
• train(bool) (:99) + should_train_index (:817-836): train=false skips model training; combined with precomputed-model params this is the shared-model distributed strategy.
• Shared-model injection points: IvfBuildParams.centroids: Option<Arc<FixedSizeListArray>> (rust/lance-index/src/vector/ivf/builder.rs:35) and PQBuildParams.codebook: Option<ArrayRef> (rust/lance-index/src/vector/pq/builder.rs:41); vector_params_have_precomputed_ivf detects injected centroids (create.rs:838-843).
• Standalone model training: lance::index::vector::ivf::build_ivf_model (rust/lance/src/index/vector/ivf.rs:1535) and build_pq_model / build_pq_model_in_fragments (rust/lance/src/index/vector/pq.rs:507,518) — these back the Java SDK's VectorTrainer (JNI wiring: java/lance-jni/src/vector_trainer.rs:107-148).
• execute_uncommitted() -> Result<IndexMetadata> (create.rs:141) — builds the segment files and returns the metadata without committing.
• IndexMetadata (rust/lance-table/src/format/index.rs:32-62): uuid, fields, name, dataset_version, fragment_bitmap, index_details (prost_types::Any), index_version, … — with a verified prost round-trip TryFrom<pb::IndexMetadata> (:139) and From<&IndexMetadata> for pb::IndexMetadata (:186). This is the segment-metadata transfer format: versioned protobuf, already the on-disk manifest encoding.
• Progress: IndexBuildProgress trait with stage_start / stage_progress / stage_complete (rust/lance-index/src/progress.rs:21-33), accepted by the builder via .progress(...) (create.rs:125).
• Commit-existing: Dataset::commit_existing_index_segments (rust/lance/src/index.rs:1441), with commit-side validation of non-empty set / unique UUIDs / non-overlapping fragment coverage (validate_segment_metadata, index.rs:100-131).
• Provenance hook: CreateIndexBuilder.transaction_properties(HashMap<String,String>) (:135) — records caller key-values into the commit's transaction file.

3.3 Verified gaps (core-side, not just FFI)

• Cancellation: no cancel/abort hook exists anywhere in the create-index path (the builder surface create.rs:51-66 has no token; grep over the path finds nothing). E4 requires core work, not just FFI.
• Listing stats: the per-segment fragment coverage exists in IndexMetadata.fragment_bitmap but is dropped by the hard-coded list JSON (§3.1). Row counts per segment are not surfaced through any current listing path.

4. Design principles

1. Thin FFI over verified core behavior — no semantic reimplementation at the C layer; every function below is a direct wrapping of a pinned core entry point.
2. lance-c house style — opaque handles; int32_t returns with lance_last_error_code() / thread-local message (lance.h:129-135); caller-frees via lance_free_string-style functions; repr(C) params structs with the established "0 = library default" convention; no silent clamping or substitution at the API boundary (existing convention, e.g. require_field in src/index.rs:459-468); Arrow C Data Interface for array exchange (already used at lance.h:59-63,709,750).
3. Additive only — no signature or JSON-shape changes to existing functions; E6 ships as a new function rather than mutating lance_dataset_index_list_json's hard-coded shape.
4. PRs are independently mergeable — each PR in the sequence is useful on its own; PR 1 is the foundational capability for distributed consumers.

5. PR 1 — distributed build loop (E1 + E2)

E1. Fragment-scoped uncommitted segment build

New opaque builder handle (the existing one-shot style cannot carry progress/cancellation state across a long build):

typedef struct LanceIndexSegmentBuilder LanceIndexSegmentBuilder;

/* Options mirror lance-c conventions: NULL/0 = default. */
typedef struct LanceIndexSegmentBuildOptions {
    /* Fragment scope: disjoint subset assigned by the coordinator.
       NULL + 0 = whole dataset (current one-shot behavior). */
    const uint32_t* fragment_ids;
    size_t          fragment_count;

    /* Caller-assigned segment UUID (16 raw bytes, RFC 4122).
       NULL = generated by the library. Coordinators SHOULD pre-assign
       so the expected segment set is known before dispatch. */
    const uint8_t*  index_uuid;

    /* Shared-model injection (vector only):
       centroids — FixedSizeList<float32>[num_partitions, dim]
       codebook  — FixedSizeList<float32>[num_sub_vectors * 2^num_bits, dim/num_sub_vectors]
       Passed via Arrow C Data Interface. Both NULL = train locally.
       When set, `train` is ignored and the injected model is used as-is. */
    struct ArrowArray*  ivf_centroids;   /* + ArrowSchema, per C Data Interface */
    struct ArrowArray*  pq_codebook;     /* + ArrowSchema */

    bool train;   /* default true; false = use injected model only, never train */
} LanceIndexSegmentBuildOptions;

LanceIndexSegmentBuilder* lance_index_segment_builder_new_vector(
    LanceDataset* dataset,
    const char* column,
    const char* index_name,                 /* NULL → "<column>_idx" */
    const LanceVectorIndexParams* params,   /* existing struct, unchanged */
    const LanceIndexSegmentBuildOptions* options
);
/* scalar twin: …_new_scalar(dataset, column, index_name, LanceScalarIndexType, params_json, options) */

/* Executes WITHOUT committing. On success returns the built segment's
   metadata as prost-encoded pb::IndexMetadata bytes (E2).
   Caller frees with lance_free_bytes(). */
int32_t lance_index_segment_builder_execute_uncommitted(
    LanceIndexSegmentBuilder* builder,
    uint8_t** out_metadata_bytes,
    size_t* out_metadata_len
);

void lance_index_segment_builder_free(LanceIndexSegmentBuilder*);
void lance_free_bytes(uint8_t*);   /* new; byte-buffer twin of lance_free_string */

Semantics (all inherited from the pinned core behavior, not re-specified):

• The build runs against the snapshot held by the open dataset handle; the returned IndexMetadata.dataset_version records it.
• Every segment carries its own UUID; duplicate-UUID and overlapping-coverage rejection happens at commit time (validate_segment_metadata, index.rs:100-131), not here.
• Segment artifacts are written beneath the dataset's _indices/<uuid>/ directory, unreferenced by any manifest until a commit lands.
• train=false never trains (should_train_index, create.rs:822-824 returns false immediately — no silent fallback). I'd additionally propose API-boundary validation: train=false without an injected model fails fast with InvalidInput, rather than relying on whatever the deeper vector build path does with neither model nor training (unspecified today — flagged, not assumed).
• Element-type and params validation identical to the existing one-shot entry points.

Standalone model training (so a coordinator/training task can produce the injected model from a sample):

/* Trains IVF centroids from a sample of the column (optionally fragment-scoped).
   out: FixedSizeList<float32> via Arrow C Data Interface. */
int32_t lance_dataset_train_ivf_centroids(
    LanceDataset* dataset, const char* column,
    uint32_t num_partitions, LanceMetricType metric,
    const uint32_t* sample_fragment_ids, size_t sample_fragment_count,
    struct ArrowArray* out_centroids, struct ArrowSchema* out_schema
);
/* PQ twin: lance_dataset_train_pq_codebook(...) */

Both wrap the pinned core entries (build_ivf_model ivf.rs:1535; build_pq_model pq.rs:507) — the same entries that back the Java VectorTrainer (vector_trainer.rs:107-148), so C and Java workers produce interchangeable models.

E2. Segment metadata serialization

The transfer format is prost-encoded pb::IndexMetadata — the same encoding the manifest uses, so it is version-tolerant by construction (round-trip pinned at format/index.rs:139,186). execute_uncommitted emits it; two helpers make it usable without a protobuf dependency:

typedef struct LanceIndexSegmentMetadata LanceIndexSegmentMetadata;

int32_t lance_index_segment_metadata_parse(
    const uint8_t* bytes, size_t len,
    LanceIndexSegmentMetadata** out
);
/* Getters: uuid (16 bytes), name, dataset_version, index_version,
   field_ids, fragment-id coverage (roaring-decoded uint32 array), index_type. */
void lance_index_segment_metadata_free(LanceIndexSegmentMetadata*);

Rationale for bytes-first: my downstream consumer ships these bytes from C++ workers to a Java coordinator and decodes them into the Java SDK's Index.Builder fields via an explicit decoder/adapter — the pinned SDK exposes only per-field setters and no public parseFrom(byte[]), so the adapter plus a version-compatibility contract is part of this work item. C-only consumers use the getters or feed the bytes straight into E5's commit. Nobody needs to re-declare the protobuf schema off-manifest.

6. PR 2 — commit-existing-segments C API (E5)

E5. Commit-existing-segments C API

Included in the current batch per maintainer feedback — a C API that can build segments but not commit them is a strange place to stop, even though my downstream coordinator commits via the Java SDK. Cross-language precedent: the pinned Python bindings already expose this loop as public LanceDataset methods — merge_existing_index_segments and commit_existing_index_segments (python/python/lance/dataset.py:4396,4402).

/* Commits previously built segments as one logical index, one new dataset version.
   Wraps Dataset::commit_existing_index_segments (rust/lance/src/index.rs:1441):
   rejects empty sets, duplicate UUIDs, overlapping fragment coverage. */
int32_t lance_dataset_commit_index_segments(
    LanceDataset* dataset,
    const char* index_name,
    const char* column,
    const uint8_t* const* segment_metadata_bytes,  /* pb::IndexMetadata array */
    const size_t* segment_metadata_lens,
    size_t segment_count,
    bool replace
);

Open question: whether physical segment merge (consolidating N segments into one — the Python merge_existing_index_segments above) should also be C-exposed. The downstream Phase-1 design explicitly does not merge; scope to be confirmed alongside the E5 PR.

7. PR 3 — build progress reporting (E3)

E3. Progress reporting

FFI bridge over the existing IndexBuildProgress trait (progress.rs:21-33):

typedef void (*LanceIndexBuildProgressFn)(
    void* user_data,
    int32_t event,          /* 0=stage_start, 1=stage_progress, 2=stage_complete */
    const char* stage,      /* borrowed; valid for the call duration */
    uint64_t total,         /* 0 = unknown (stage_start only) */
    const char* unit,       /* borrowed; e.g. "partitions", "rows" */
    uint64_t completed      /* stage_progress only */
);
/* set on the builder: */
void lance_index_segment_builder_set_progress(
    LanceIndexSegmentBuilder*, LanceIndexBuildProgressFn, void* user_data
);

Threading contract (to be documented on the function): callbacks fire from lance-c's internal tokio runtime threads; they must be thread-safe and non-blocking. This mirrors how block_on drives all current index calls (lance-c src/runtime.rs).

8. PR 4 — per-segment coverage/stats in listing (E6)

E6. Per-segment coverage/stats in listing

New additive function (the existing list JSON keeps its hard-coded shape, §4.3):

/* One entry per segment:
   {name, uuid, columns, type, dataset_version, index_version,
    fragment_ids:[...], created_at (when recorded)} */
const char* lance_dataset_index_segments_json(const LanceDataset* dataset);

Row counts per segment are not promised: they are not surfaced by any current listing path (§3.3) and would need core plumbing — stated plainly so downstream consumers don't plan against them.

9. Separate track — cooperative cancellation (E4)

E4. Cooperative cancellation

Honest boundary: there is no cancellation hook in the core create path today (§3.3). Per maintainer feedback, E4 is unblocked separately from the PR sequence: the core cancellation token lands in lance first (rust/lance/src/index/create.rs), and the lance-c side is only the FFI bridge. Proposal, to be settled with maintainers before implementation:

1. Core: add a lightweight CancellationToken-style parameter to CreateIndexBuilder, checked at stage boundaries (per-partition build loop, shuffle batches, training iterations) — the same points that already drive stage_progress.
2. lance-c: LanceIndexBuildCancelToken opaque handle; lance_index_segment_builder_set_cancel_token(builder, token); lance_index_build_cancel(token) from any thread.
3. Semantics: cancellation is cooperative and best-effort; an aborted build returns a dedicated error code (LANCE_ERR_CANCELLED), leaves unreferenced segment files for normal Lance cleanup, and never commits.

Alternative considered: kill via dropping the runtime future. Rejected — unsafe mid-I/O and unobservable; cooperative checkpoints are the only behavior I can specify honestly.

10. Semantics contract for distributed consumers

The invariants this proposal relies on, each pinned to core behavior:

1. Coordinator pre-assigns per-worker UUIDs (index_uuid), so the expected segment-UUID set is deterministic before dispatch; commit rejects duplicates (index.rs:100-131).
2. Fragment coverage is explicit and validated: workers build exactly their assigned disjoint subsets (fragments, create.rs:104); commit rejects overlap at commit time.
3. Shared-model strategy: a training task derives centroids + codebook once (E1 training entries); workers inject them and build with train=false; vector_params_have_precomputed_ivf (create.rs:838-843) detects the precomputed path. Segments built this way remain merge-compatible and give uniform fan-out recall; per-segment independently-trained models are valid for fan-out query but preclude later merge.
4. Version pinning: each segment records the dataset version it was built from; commits are normal Lance transactions — concurrent appends are compatible, conflicting operations surface as retryable commit conflicts per the format's transaction semantics.
5. Failure hygiene: uncommitted segment files are unreferenced artifacts, reclaimed by standard Lance cleanup; nothing here introduces a new commit mode.

11. Compatibility and versioning

• All additions are new symbols + one new free function (lance_free_bytes); no existing signature, struct layout, or JSON shape changes. lance-c is 0.x; I'd still treat the additions as stable-once-merged.
• pb::IndexMetadata bytes inherit the manifest's protobuf evolution rules (field additions only, readers ignore unknowns) — the transfer format does not need its own versioning.

12. Test plan (per PR)

• PR 1 (E1/E2): fragment-scoped uncommitted build returns metadata whose decode matches assignment (UUID, fragment set, dataset_version, index_details present); injected centroids+codebook path (train=false) produces segments queryable after an E5/Java-SDK commit; train=false without model → InvalidInput; scalar (BTREE/BITMAP) fragment-scoped builds; artifacts land under _indices/<uuid>/ and manifests are untouched pre-commit.
• PR 2 (E5): multi-segment commit = single version increment; duplicate UUID / overlapping coverage rejected; replace semantics on same-name commit.
• PR 3 (E3): callback fires stage_start/progress/complete in order from runtime threads; NULL callback = no-op.
• PR 4 (E6): segment JSON carries fragment_ids matching each segment's fragment_bitmap; legacy index_list_json output byte-identical to before.
• E4 (separate track): cancel before/during each stage → LANCE_ERR_CANCELLED, no commit, no dangling manifest references.
• Cross-SDK: C-built segment metadata bytes → Java decoder/adapter → Index.Builder reconstruction → commitExistingIndexSegments → k-NN equivalence vs single-shot build (the underlying build→commit→consume loop is already PoC-verified Java-to-Java on 9.1.0-beta.3).

13. Questions for maintainers — answered

All six answered by @jja725 in this comment ("please send the PRs"); the outcomes are folded into the plan above. Original questions, answers in brief:

1. Wave split — is landing Wave 1 (E1+E2) alone acceptable, with E3–E6 as follow-up PRs? → Yes — and finer is welcome. The plan is now a four-PR sequence with E4 split out as a separate track.
2. Builder handle vs one-shot — I chose an opaque builder because progress/cancellation need cross-call state; does that fit the roadmap, or do you prefer a single call with an options struct? → Builder handle agreed — but drop struct_size: no precedent in the header, and the 0.x ABI is unstable on purpose (consistent versioning can wait for 1.0). Removed from E1.
3. Arrow C Data Interface for model injection — acceptable carrier for centroids/codebook, or do you prefer flat float* + shape params? → Arrow C Data Interface is good.
4. E4 approach — token-in-builder checked at stage boundaries, as proposed? Any preference on where checkpoints live in lance-index? → Unblock this one separately; E4 now runs as its own track (§9).
5. E5 scope — include commit-existing now, or defer until a C-only consumer asks? And is physical segment merge something you want exposed at all? → Include E5 now (it is PR 2) — a C API that builds segments but cannot commit them is a strange place to stop. Merge exposure scoped alongside the E5 PR.
6. Contribution mode — I intend to implement these as upstream PRs. Preferred venue: this issue per wave, or one tracking issue? → This issue is the tracker; one PR per extension, or per wave when two extensions are genuinely one change; each PR references this issue.

---

Prepared against lance-c v0.1.6 (98e4d54) and lance 9.1.0-beta.3; paths and line numbers will be re-verified against then-current master when each PR is submitted. Downstream: apache/doris#66497.

[u70b3]

Follow-up after downstream review in apache/doris#66497 (thanks @Gabriel39 for the sharp read):

Decoupling confirmed. Doris 4.2 is scoped to the pinned lance-c 0.1.2 one-shot APIs; nothing in this RFC gates that release — this issue is the post-4.2 distributed track. The waves remain independently mergeable, and I'm glad to implement them in whatever order and shape maintainers prefer, at your pace. The issue body has been updated to say this explicitly (the earlier "blocking" framing is gone).

E2 correction folded into the body. The pinned Java SDK's Index.Builder exposes only per-field setters and no public parseFrom(byte[]), so pb::IndexMetadata bytes are not directly consumable cross-SDK. The E2 transfer story now explicitly includes a Java decoder/adapter plus a version-compatibility contract; C-only consumers (the getters, E5 commit) are unaffected.

Implementation commitment unchanged.

[jja725]

Thanks for writing this up properly, and yes, please send the PRs. Open source contribution are always welcomed.

Answers to your questions.

1. Wave split. Yes. If anything I'd go finer than you proposed.

2. Builder handle. Agreed.

One thing on the options struct: skip struct_size. We don't use it anywhere in the header today, and Cargo.toml has versioning = false with a note that 0.x ABI is unstable on purpose. Adding it to exactly one struct creates a convention we'd then have to either honor everywhere or explain forever. At 1.0 we can do that consistently.

3. Arrow C Data for centroids and codebook. Arrow is good

4. Cancellation. let's unblock this one separately

5. E5 and merge. We can include E5 now. It's FFI over a function that already validates itself, and shipping a C API that can build segments but not commit them is a strange place to stop even if your own coordinator is Java.

6. Contribution mode. Keep this issue as the tracker, one PR per extension, or per wave when two extensions are genuinely one change. Reference the issue from each PR and we'll maintain the checklist here.

[u70b3]

Thanks @jja725 for the point-by-point answers — all adopted, and the issue body is updated to match:

• Finer split: PR 1 = E1+E2 (genuinely one change), PR 2 = E5, PR 3 = E3, PR 4 = E6.
• struct_size removed from LanceIndexSegmentBuildOptions; no per-struct versioning until a consistent 1.0 convention.
• Arrow C Data Interface confirmed as the centroids/codebook carrier.
• E5 included in the current batch; physical segment merge exposure will be scoped alongside the E5 PR.
• One PR per extension (E1+E2 as a single PR), each referencing this issue as the tracker.

E4 proceeds as a separate track: the core token work belongs in lance-format/lance (rust/lance/src/index/create.rs), with lance-c contributing only the FFI bridge. When we get there — would you prefer a dedicated tracking issue in the main lance repo, or keep tracking it here?

I'll start with PR 1 (E1+E2).

[u70b3]

Implementation progress

Draft PR opened for the foundational distributed-build slice (E1 + E2): #57.

Completed in PR 1

• Upgrade the pinned Lance core baseline to 9.1.0-beta.3 and migrate affected APIs.
• Add snapshot-owned, single-use scalar and vector uncommitted segment builders.
• Support fragment-scoped builds and explicit segment UUIDs where supported by Lance core.
• Add standalone reusable IVF centroid training.
• Add residual PQ codebook training tied to the corresponding IVF model.
• Add reusable Arrow C Data model injection for multiple disjoint fragment segments.
• Add protobuf IndexMetadata encode/decode plus C metadata getters.
• Add C++ move-only RAII wrappers.
• Cover multi-fragment, NULL/empty, malformed Arrow, sliced/null model, lifecycle, overflow, and real C/C++ dynamic-link cases.
• Add model provenance validation for metric, dimension, PQ parameters, and IVF identity.

Follow-up work

• PR 2 / E5: commit existing segment metadata as one logical index, including duplicate UUID and overlapping-coverage validation; decide physical segment merge exposure.
• PR 3 / E3: bridge IndexBuildProgress callbacks with documented threading/lifetime rules.
• PR 4 / E6: add per-segment coverage/details listing without changing the legacy JSON shape.
• Separate E4 track: add cooperative cancellation in Lance core first, then expose the token through lance-c.
• Cross-SDK follow-up: decode C-produced metadata in the Java coordinator and verify commit/query equivalence.

The draft PR intentionally does not close this tracker; the remaining extensions stay tracked here.