fix: harden DiskANN against type confusion and invalid vector datums by mostafa · Pull Request #280 · timescale/pgvectorscale

mostafa · 2026-08-12T10:24:47Z

Summary

Resolve pgvector's extension schema from the catalogs and create DiskANN opclasses with safely quoted, schema-qualified pgvector type/operator references while preserving PostgreSQL's standard extension search_path.
Add a 0.9.0-to-0.9.1 upgrade script that rebinds C entry points and aborts if an existing vector opclass is bound incorrectly.
Validate signed typmod, persisted metapage dimensions, and detoasted datum size/dimension before constructing a slice.
Implement amvalidate for the vector DiskANN operator classes.

Vulnerability

DiskANN operator-class SQL used unqualified vector/operators, so a pre-created type in the extension target schema could capture the binding. Combined with unvalidated typmod-to-dimension conversion and using the embedded dim as a slice length without a size check, attacker-controlled bytes could reach native DiskANN paths, causing a backend crash, memory disclosure, or out-of-bounds write. Further privilege-escalation impact was reported for the detoast path.

Root cause

Install SQL did not bind pgvector-owned type/operator names to the pgvector extension schema.
Native code assumed a valid typmod, internally consistent metapage dimensions, and a well-formed pgvector datum layout.

Why this change is safe

Fresh installations bind directly to pgvector-owned objects even when the vectorscale target schema contains shadow objects.
Upgrades from 0.9.0 reject poisoned opclasses transactionally before the new extension version becomes active.
Legitimate vector(N) indexes retain positive typmods and matching datum layouts, so the native checks are no-ops for supported configurations.

Potential side effects

An upgrade with an incorrectly bound opclass fails until the affected objects are dropped and recreated.
Columns without a valid vector(N) typmod can no longer create DiskANN indexes.

Validation

Multi-version CI covers PostgreSQL 14 through 18 on amd64 and arm64.
Integration coverage verifies that a target-schema shadow type cannot capture fresh-install opclasses and that typmod-less vector indexes fail cleanly.
Unit coverage verifies invalid typmods, metapage dimensions, malformed datum layouts, dimension mismatches, and amvalidate.

Bind vector DiskANN operator classes with @extschema:vector@ so extension install/upgrade cannot pick up a schema-shadowed non-vector type, and fail loudly if an existing opclass is already bound wrongly.

Require pgvector's vector type before index build/open, reject negative or oversized typmods and inconsistent metapage dimensions, and check detoasted datum size against the embedded dimension before constructing any slice. Also implement amvalidate for the vector DiskANN operator classes.

Copilot

Pull request overview

Hardens DiskANN against schema-shadowed vector types and malformed vector/index metadata.

Changes:

Schema-qualifies pgvector operator classes and validates bindings.
Adds index type, typmod, metapage, and datum-layout checks.
Implements operator-class validation.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
`pgvectorscale/src/access_method/vector_type.rs`	Adds vector OID and layout validation.
`pgvectorscale/src/access_method/pg_vector.rs`	Validates detoasted vector data.
`pgvectorscale/src/access_method/mod.rs`	Hardens opclasses and `amvalidate`.
`pgvectorscale/src/access_method/meta_page.rs`	Validates index metadata on creation/open.
`pgvectorscale/src/access_method/build.rs`	Validates index type and typmod.
`pgvectorscale/sql/vectorscale--0.8.0--0.9.0.sql`	Hardens upgrade opclasses.
`pgvectorscale/sql/vectorscale--0.7.1--0.8.0.sql`	Hardens upgrade opclasses.
`pgvectorscale/sql/vectorscale--0.7.0--0.7.1.sql`	Hardens upgrade opclasses.
`pgvectorscale/sql/vectorscale--0.6.0--0.7.0.sql`	Hardens upgrade opclasses.
`pgvectorscale/sql/vectorscale--0.5.1--0.6.0.sql`	Hardens upgrade opclasses.
`pgvectorscale/sql/vectorscale--0.5.0--0.5.1.sql`	Hardens upgrade opclasses.
`pgvectorscale/sql/vectorscale--0.4.0--0.5.0.sql`	Hardens upgrade opclasses.
`pgvectorscale/sql/vectorscale--0.3.0--0.4.0.sql`	Hardens upgrade opclasses.
`pgvectorscale/sql/vectorscale--0.2.0--0.3.0.sql`	Hardens upgrade opclasses.
`pgvectorscale/sql/vectorscale--0.0.2--0.2.0.sql`	Hardens upgrade opclasses.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Make opclass binding portable across supported PostgreSQL versions, reject stale/malformed type metadata, and add upgrade and regression coverage.

Keep option parsing coverage compatible with the new invariant that indexed dimensions cannot exceed the vector's full dimension.

Resolve pgvector objects through safely quoted dynamic SQL while retaining PostgreSQL's standard extension execution context.

Rely on qualified fresh-install bindings and the blocking 0.9.0 upgrade validation instead of runtime checks that cannot observe supported poisoned states.

svenklemm · 2026-08-15T07:11:53Z


        -- Register the operator with the system catalogs for proper selectivity estimation
        -- This is done by adding entries to pg_amop for the array_ops operator class
        EXECUTE format(


Suggested change

EXECUTE pg_catalog.format(

unqualified function call

svenklemm · 2026-08-15T07:31:11Z

        SELECT 1 FROM pg_catalog.pg_operator 
        WHERE oprname = '&&' 
        AND oprleft = 'smallint[]'::regtype 
        AND oprright = 'smallint[]'::regtype


Suggested change

SELECT 1 FROM pg_catalog.pg_operator

WHERE oprname = '&&'

AND oprnamespace = '@extschema@'::regnamespace

AND oprleft = 'smallint[]'::regtype

AND oprright = 'smallint[]'::regtype

missing schema classfiier

svenklemm · 2026-08-15T07:50:57Z

Keep in mind that amvalidate is not called by postgres to validate, but instead it is used in testing/has to be called explicitly.

svenklemm · 2026-08-15T08:26:54Z

+DO $$
+DECLARE
+    expected_vector_type oid;
+BEGIN
+    SELECT t.oid
+    INTO STRICT expected_vector_type
+    FROM pg_catalog.pg_extension e
+    JOIN pg_catalog.pg_type t
+      ON t.typnamespace = e.extnamespace
+     AND t.typname = 'vector'
+    WHERE e.extname = 'vector';
+
+    IF EXISTS (
+        SELECT 1
+        FROM pg_catalog.pg_opclass c
+        JOIN pg_catalog.pg_am am ON am.oid = c.opcmethod
+        WHERE am.amname = 'diskann'
+          AND c.opcnamespace = (
+              SELECT oid
+              FROM pg_catalog.pg_namespace
+              WHERE nspname = '@extschema@'
+          )
+          AND c.opcname IN (
+              'vector_cosine_ops',
+              'vector_l2_ops',
+              'vector_ip_ops'
+          )
+          AND c.opcintype IS DISTINCT FROM expected_vector_type
+    ) THEN
+        RAISE EXCEPTION
+            'diskann: a vector operator class is not bound to pgvector''s vector type; drop the affected operator class and recreate the extension objects';
+    END IF;
+END;
+$$;


looks like this doesnt catch all variants:

CREATE SCHEMA real_vector; CREATE EXTENSION vector WITH SCHEMA real_vector; CREATE SCHEMA evil; CREATE FUNCTION evil.fake_dist(real_vector.vector, real_vector.vector) RETURNS float8 LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ SELECT 0::float8 $$; CREATE OPERATOR evil.<=> ( LEFTARG = real_vector.vector, -- genuine RIGHTARG = real_vector.vector, -- genuine FUNCTION = evil.fake_dist); CREATE EXTENSION vectorscale VERSION '0.9.0' WITH SCHEMA evil;

mostafa added 3 commits August 12, 2026 12:18

fix: qualify DiskANN opclasses to pgvector's vector type

2964672

Bind vector DiskANN operator classes with @extschema:vector@ so extension install/upgrade cannot pick up a schema-shadowed non-vector type, and fail loudly if an existing opclass is already bound wrongly.

style: apply cargo fmt

3bee40e

mostafa requested review from erimatnor and a balanced review from Copilot August 12, 2026 10:34

Copilot started reviewing on behalf of mostafa August 12, 2026 10:38 View session

Copilot AI reviewed Aug 12, 2026

View reviewed changes

mostafa added 2 commits August 12, 2026 14:00

fix: complete DiskANN type safety hardening

da2c7b5

Make opclass binding portable across supported PostgreSQL versions, reject stale/malformed type metadata, and add upgrade and regression coverage.

test: use valid indexed dimensions in option coverage

983a48d

Keep option parsing coverage compatible with the new invariant that indexed dimensions cannot exceed the vector's full dimension.

mostafa marked this pull request as ready for review August 12, 2026 13:24

mostafa requested a review from a team as a code owner August 12, 2026 13:24

erimatnor reviewed Aug 13, 2026

View reviewed changes

mostafa added 2 commits August 13, 2026 16:47

fix: preserve extension script search path

8e81a6f

Resolve pgvector objects through safely quoted dynamic SQL while retaining PostgreSQL's standard extension execution context.

refactor: remove redundant index type checks

d726955

Rely on qualified fresh-install bindings and the blocking 0.9.0 upgrade validation instead of runtime checks that cannot observe supported poisoned states.

svenklemm reviewed Aug 15, 2026

View reviewed changes

Uh oh!

Conversation

mostafa commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vulnerability

Root cause

Why this change is safe

Potential side effects

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

svenklemm Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

svenklemm Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

svenklemm Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svenklemm Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

svenklemm commented Aug 15, 2026

Uh oh!

svenklemm Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mostafa commented Aug 12, 2026 •

edited

Loading

svenklemm Aug 15, 2026 •

edited

Loading