Skip to content

fix: hold meta-page lock across reinit and writes in MetaPage::store - #270

Draft
pavanmanishd wants to merge 1 commit into
timescale:mainfrom
pavanmanishd:fix/264-parallel-build-meta-race
Draft

fix: hold meta-page lock across reinit and writes in MetaPage::store#270
pavanmanishd wants to merge 1 commit into
timescale:mainfrom
pavanmanishd:fix/264-parallel-build-meta-race

Conversation

@pavanmanishd

Copy link
Copy Markdown

Summary

Parallel DiskANN index builds can fail with assertion 'left == right' failed (see #264). Every parallel worker calls MetaPage::store(false) from finalize_index_build (build.rs:940), and store() released the exclusive buffer lock between reinit and each subsequent write. Concurrent workers could interleave the operations and produce the observed offset mismatch.

This PR restructures MetaPage::store to hold one WritablePage across reinit and both writes, committing once at the end. Concurrent callers now serialize on the Postgres buffer lock.

Note for reviewers: the root cause is slightly different from what's described in #264 — the MetaV1 branch of fetch() is not the trigger. Freshly built indexes use PageType::Meta, and fetch() on that branch is read-only (Self::load). The race I'm seeing is in MetaPage::store itself, exercised from finalize_index_build in each parallel worker.

Changes

  • pgvectorscale/src/access_method/meta_page.rsstore() now acquires a single WritablePage (via new for first_time=true, or modify + in-place reinit for first_time=false), writes header + meta, and commits once.
  • pgvectorscale/src/util/chain.rs — new write_chain_item_to_page(page, data) helper for the single-item, caller-holds-lock case (keeps the ChainItemHeader serialization format in one module). Removed the now-unused ChainTapeWriter::reinit.

Net diff: +33 / −34 lines across 2 files.

Test plan

  • cargo pgrx install --release --features pg18 builds cleanly; no new clippy warnings on the changed files.
  • Reproduced the race on main with a 50 ms sleep inserted between reinit and the first write in store(): 4 of 5 iterations fail with the exact assertion from [Bug]: <Title>Parallel_Build_Assertion_Failure #264.
  • Verified the fix — with the patch applied and a 50 ms sleep inserted inside the single lock window to stress serialization: 5 of 5 iterations pass.
  • cargo pgrx test pg{17,18} — to be confirmed by CI (local run blocked by a pgrx --sudo install quirk on macOS, unrelated to this change).

Repro script preserved for convenience:

DROP TABLE IF EXISTS v;
CREATE TABLE v (id SERIAL PRIMARY KEY, embedding vector(64));
INSERT INTO v (embedding)
SELECT (SELECT array_agg(random()::real) FROM generate_series(1, 64))::vector
FROM generate_series(1, 50000);

SET diskann.min_vectors_for_parallel_build = 100;
SET max_parallel_maintenance_workers = 16;
SET diskann.force_parallel_workers = 16;

DO $$ BEGIN FOR i IN 1..5 LOOP
    BEGIN
        EXECUTE 'DROP INDEX IF EXISTS v_diskann_idx';
        EXECUTE 'CREATE INDEX v_diskann_idx ON v USING diskann (embedding vector_l2_ops)
                 WITH (storage_layout = ''memory_optimized'')';
        RAISE NOTICE 'Iteration % succeeded', i;
    EXCEPTION WHEN others THEN
        RAISE NOTICE 'Iteration % FAILED: %', i, SQLERRM;
    END;
END LOOP; END $$;

Notes

I intentionally did not add an in-tree regression test because reproducing the bug deterministically requires injecting a sleep inside store(). Happy to add a #[cfg(test)]-gated race-widener test on request.

Fixes #264

Parallel index-build workers all call MetaPage::store(false) from
finalize_index_build, which released the exclusive buffer lock
between reinit and each subsequent write. Concurrent workers could
interleave reinit/write and trigger the "offset N != 1" assertion
failure on block 0.

Restructure store() to hold one WritablePage across reinit, header
write, and meta write, committing once at the end. Extract
write_chain_item_to_page helper for the single-item,
caller-holds-lock case so the chain-item serialization stays in
util/chain.rs. Remove the now-unused ChainTapeWriter::reinit.

Fixes timescale#264
@CLAassistant

CLAassistant commented Apr 22, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: <Title>Parallel_Build_Assertion_Failure

2 participants