Skip to content

Track progress of CREATE INDEX via pg_stat_progress_create_index. - #275

Open
andreheringer wants to merge 3 commits into
timescale:mainfrom
andreheringer:agh/index-bluid-progress
Open

Track progress of CREATE INDEX via pg_stat_progress_create_index.#275
andreheringer wants to merge 3 commits into
timescale:mainfrom
andreheringer:agh/index-bluid-progress

Conversation

@andreheringer

Copy link
Copy Markdown

Summary

Report diskann CREATE INDEX progress via pg_stat_progress_create_index for both sequential and parallel builds, including per-worker live tuple counts.

Changes

  • Sequential build: tuples_total is set to the heap relation's reltuples, and tuples_done is updated every 1000 tuples inside build_callback_internal. The existing subphase transitions ("training quantizer" -> "building graph" -> "finalizing graph") are preserved.
  • Parallel build: the blocking WaitForParallelWorkersToFinish on the leader is replaced with an interruptible observer loop. Workers periodically flush local_ntuples into a per-worker slot in DSM (allocated via a new ParallelSharedLayout wrapper). The leader sums the slots and updates tuples_done every 500ms via WaitLatch, exiting when all workers signal workers_completed.
  • Progress parameter slots changed to the PG16 standard positions (subphase = 10, total = 11, done = 12).
  • Python regression tests added in tests/test_progress_reporting.py (test_progress_increases, test_progress_reaches_completion).

Testing:

  • cargo pgrx test (pg16): 82 passed, 10 ignored, 0 failed
  • pytest tests/: 10 passed, 0 failed
  • Manual live-progress probe against a freshly built pgvectorscale-debug image confirmed tuples_done rising 1024 -> 27060 during the build of a 30k-row SbqCompression index with 4 workers.

Fixes #122

Parallel progress report to the main PID is done by pulling a pool of
processed tuples done by each worker, the loop executes until all
workers are oberved and a final barrier was kept for safety
@andreheringer
andreheringer requested a review from a team as a code owner July 7, 2026 06:20
@CLAassistant

CLAassistant commented Jul 7, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose indexing progress via pg_stat_progress_create_index

2 participants