support deletes for leading segment merges

[sam-herman]

Description

For leading segment merges #133 we made delete an exception that will force full reconstruction of the graph.
This change removes that exception and allow for deletes to also happen in-place on the leading segment, even if those deletes happen on the leading segment itself.

This allows for the results previously shown for the incremental insertion into existing graph change #167 to also shown during deletes.

Benchmarks

Latest delete benchmark using the delete script shows various deletion vs recall scenarios and confirms that jVector behaves in a FreshDiskANN manner.

Related Issues

Resolves #171

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• [ x] Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[akash-shankaran]

Is it worth putting this feature behind a feature flag config or experimental tag?
To avoid any regression that may have slipped through.

[sam-herman]

Is it worth putting this feature behind a feature flag config or experimental tag? To avoid any regression that may have slipped through.

Right now this will only kick in if we modify the minimum batch for quantization parameter. If we keep it as is, should be no difference and this functionality of leading segment merges will not be triggered.

[sam-herman]

I see test failure of timeout, but it doesn't seem to be related to the change as it's not even touching the merge code yet:

InternalKNNEngineTests > testQuantizationWithOverQueryParameter FAILED
    java.net.SocketTimeoutException: 30000 MILLISECONDS
        at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:1326)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:371)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:374)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:374)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:374)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:359)
        at org.opensearch.knn.index.engine.CommonTestUtils.bulkAddKnnDocs(CommonTestUtils.java:314)
        at org.opensearch.knn.index.engine.CommonTestUtils.bulkAddKnnDocs(CommonTestUtils.java:280)
        at org.opensearch.knn.index.engine.CommonTestUtils.bulkAddKnnDocs(CommonTestUtils.java:267)
        at org.opensearch.knn.index.engine.InternalKNNEngineTests.testQuantizationWithOverQueryParameter(InternalKNNEngineTests.java:530)

This seems to indicate that the in-memory cluster is super slow and timing out when adding new documents to the index for some reason.
it does pass locally, and seem to pass fine in some environments but sporadically fail on others.

I would recommend that we make changes for this test to share in-memory cluster across tests (switch to @suite mode) and hopefully it will be more stable on GH. Or alternatively see if the next version ugprade fixes it, as it seems to have been introduced after latest upgrade.

what do you think? @akash-shankaran

[akash-shankaran]

I would recommend that we make changes for this test to share in-memory cluster across tests (switch to @suite mode) and hopefully it will be more stable on GH. Or alternatively see if the next version ugprade fixes it, as it seems to have been introduced after latest upgrade.

what do you think? @akash-shankaran

I agree. This failure is unrelated to your change, lets leave it for now and tackle it after the 3.3 upgrade. This test has also failed (only on GH servers) during the 3.3 upgrade PR. Failure from the upgrade PR (same test) for reference:
https://github.com/opensearch-project/opensearch-jvector/actions/runs/19356571505/job/55378896279?pr=201

I'll create a GH issue to track this, if the next upgrade doesn't fix it. .