Skip to content

add vector search benchmarks - #311

Open
vkarpov15 wants to merge 4 commits into
mainfrom
vkarpov15/benchmarking
Open

add vector search benchmarks#311
vkarpov15 wants to merge 4 commits into
mainfrom
vkarpov15/benchmarking

Conversation

@vkarpov15

Copy link
Copy Markdown
Collaborator

What this PR does:

Basic benchmarks for vector search:

  1. writeVector: insert 21k vector documents in batches of 20 using insertMany with ordered: false
  2. queryVector: execute 2000 ANN queries 10 at a time on the 21k vectors inserted

21k documents just have a text and vector property. Paradoxically, it seems like collections are actually faster than tables in this case. A big reason why collections are faster though is because collections project out the vector field by default, it seems like nearly 90% of the overhead of tables vs collections is eliminated by projecting out the vector field on the read path.

Tables

  • Tables insert: {"inserted":21347,"seconds":376.129264,"docsPerSecond":56.754425,"secondsPerBatch":0.352395}
  • Tables read: {"queries":2000,"parallelism":10,"seconds":55.601865,"queriesPerSecond":35.970016,"secondsPerBatch":0.278009}
  • Tables read w/ vector projected out: {"queries":2000,"parallelism":10,"seconds":32.994885,"queriesPerSecond":60.615457,"secondsPerBatch":0.164974}

Collections

  • Collections insert: {"inserted":21347,"seconds":222.619178,"docsPerSecond":95.890211,"secondsPerBatch":0.208572}
  • Collections read: {"queries":2000,"parallelism":10,"seconds":28.87107,"queriesPerSecond":69.273497,"secondsPerBatch":0.144355}

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Changes manually tested
  • Automated Tests added/updated
  • Documentation added/updated
  • CLA Signed: DataStax CLA

@vkarpov15
vkarpov15 requested a review from a team as a code owner January 26, 2026 20:09
Copilot AI review requested due to automatic review settings January 26, 2026 20:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds basic vector search benchmark scripts to compare insert and query performance between Astra tables and collections.

Changes:

  • Introduces benchmarks/writeVector.mjs to create/sync a content table or collection, then bulk-insert ~21k vector documents in batches and log throughput.
  • Introduces benchmarks/queryVector.mjs to run 2,000 parallelized ANN-style vector queries against the content model and log query performance metrics.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File Description
benchmarks/writeVector.mjs Sets up the Content schema and collection/table, then bulk-inserts vectorized movie documents and prints insert performance stats.
benchmarks/queryVector.mjs Uses the existing Content schema to perform repeated vector similarity queries and prints query performance stats.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread benchmarks/writeVector.mjs Outdated
Comment thread benchmarks/queryVector.mjs Outdated
Comment thread benchmarks/queryVector.mjs
Comment thread benchmarks/writeVector.mjs Outdated
Comment thread benchmarks/writeVector.mjs Outdated
vkarpov15 and others added 3 commits January 27, 2026 12:46
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@vkarpov15

Copy link
Copy Markdown
Collaborator Author

I checked and confirmed that writing $binary doesn't meaningfully help the writeVector case

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants