Fix handling of deletes during leading segment merge

[ashkrisk]

Description

This PR fixes the issues with the leading segment merge logic which show up when delete operations are involved. A set of test cases are added which fail in the absence of the changes in this PR (prior to the revert).

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ashkrisk]

I'm mostly done with the main code, barring a few tweaks. I'm still planning to add better tests.

[ashkrisk]

This PR is essentially done, but there are conflicts because of the revert that was just merged.

[akash-shankaran]

This PR is essentially done, but there are conflicts because of the revert that was just merged.

Hey @ashkrisk I did mention this to Ted yesterday. If you need to revive the change in your PR, please feel free to do so. Let us know when we can review this.

[ashkrisk]

• The first commit in this PR is an "undo revert" commit, reverting Revert of support deletes for incremental insertion #240.
• The second commit is the actual fix and test.
• The third commit removes some of the unnecessary changes in the "undo revert" so that the splash zone is lowered, hopefully making it easier to review as a whole.

[akash-shankaran]

how do we disable this feature in production if we had to? Is it possible to enable/disable it via a config?

This is to avoid a repeat of what happened in our scale test environment.

[ashkrisk]

Leading segment merge is automatically disabled once the number of live vectors involved the merge crosses the threshold for using PQ. One way to turn it off completely is to set a PQ threshold below the minimum segment size.

Note that the error you saw in the test environment wasn't because of leading segment merge per se, rather because of a bug in the way that ord <-> docId mappings were being saved. The bug was introduced as part of the changes to support deletions in the context of leading segment merge.

[akash-shankaran]

Note that the error you saw in the test environment wasn't because of leading segment merge per se, rather because of a bug in the way that ord <-> docId mappings were being saved. The bug was introduced as part of the changes to support deletions in the context of leading segment merge.

Thanks for alleviating my concern here. Yes, understand the bug was due to the ordinal mappings, which your PR addresses.
My concern is with the deletes path running into any unforeseen issues in production, and us having no way to disable the feature (i.e. incremental graph build during deletes). If there is a way to safely/enable this feature via a config (fall back to full graph rebuild), that would be great. We can remove the config after a couple of releases, when no issues are reported.
Please also let me know if you think this is unnecessary, and won't happen after your fix.

[ashkrisk]

I added a configuration parameter advanced.leading_segment_merge_disabled which can be set to true to skip leading segment merge entirely.

Note that I added this by looking at how the other parameters were configured. It should be enough to allow the configuration to be set in Opensearch and propagated to the plugin, and the test in JVectorEngineIT seems to recognize it, but let me know if that's not the case.

[opensearch-trigger-bot]

The backport to 3.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-3.3 3.3
# Navigate to the new working tree
cd .worktrees/backport-3.3
# Create a new branch
git switch --create backport/backport-242-to-3.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f33bb417ae2a63d5902426bbf0f8d7aa07d80e7d
# Push it to GitHub
git push --set-upstream origin backport/backport-242-to-3.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-3.3

Then, create a pull request where the base branch is 3.3 and the compare/head branch is backport/backport-242-to-3.3.