Enable Intel ScalableVectorSearch support

[dian-lun-lin]

Pull Request resolved: #4450

This pull request introduces support for Intel ScalableVectorSearch, integrating Intel's proprietary LVQ and LeanVec technologies in binary form. The following index types are now supported: IndexSVSVamana, IndexSVSVamanaLVQ, IndexSVSVamanaLeanVec, and IndexSVSFlat. IndexSVSVamana and IndexSVSFlat utilizes SVS open-source float32/float16/int8 implementation.

Key features and enhancements include:

• Implemented search, add, and remove_ids
• Implemented filter search and range_search
• Enabled SVS open-source fp16 and scalar quantization implementation
• Enabled factory methods
• Enabled save/load
• Enabled Python bindings
• Enable fallback mechanism that fallbacks to 8-bit scalar quantization if LVQ/LeanVec is used on non-intel hardware CPUs
• Added examples in both Python and C++ under the tutorial/ directory
• Added FAISS_ENABLE_SVS flag to allows users to optionally enable SVS

TODOs:

• Support more indices such as IndexSVSIVF and IndexSVSFlat with LVQ and LeanVec
• Documentation

[ahuber21]

Hi @mnorris11 thanks for the feedback. I addressed all open comments.
We are still working on adding the runtime flag to signal LVQ/LeanVec support. This will be added shortly to the IndexSVSVamana as discussed.
In the meantime, we'll be happy to address other concerns about implementation details, tests, or performance you may have.

One more question: Do you want to enable the SVS build plus the new tests in your CI? Should we do that as part of this PR?

[mnorris11]

Sure, feel free to add in this PR. Code pointers:

• https://github.com/facebookresearch/faiss/blob/main/.github/actions/build_cmake/action.yml#L3-L6 add new parameter here like svs similar to the current cuvs or rocm. Update the cmake command in this file to pass the right value of svs.
• https://github.com/facebookresearch/faiss/blob/main/.github/workflows/build-pull-request.yml Can add 1 new entry here that needs: linux-x86_64-cmake just like the others (can check AVX2 as an example). Pass some new parameter like svs as true.
• Then when we run the workflows, we should see it in the list here.

[ahuber21]

@mnorris11 test_svs.cpp and test_svs.py should be executed on the next run. The new step is defined here

https://github.com/ahuber21/faiss/blob/2af4cf1fecc13249ba052957ac2e51b729aca11c/.github/workflows/build-pull-request.yml#L147-L157

[alexanderguzhva]

@ahuber21

• FAISS_ENABLE_SVS is not mentioned in INSTALL.md document

[alexanderguzhva]

Also, it would be nice to have an instruction on how to install SVS. Or whether a user needs to do anything about it, package, distrubute, etc. I see that currently it is downloaded locally if you build the Faiss from the source. But the directory with a built faiss library (say, {FAISS_SRC}/build/faiss with the libfaiss_avx512_spr.so shared library file inside) does not contain any SVS-related files.

A quick how-to instruction would be appreciated by users, I suppose.

[alexanderguzhva]

terminate called after throwing an instance of 'faiss::FaissException'
what(): Error in virtual void faiss::IndexSVSVamana::search(faiss::idx_t, const float*, faiss::idx_t, float*, faiss::idx_t*, const faiss::SearchParameters*) const at /home/ubuntu/faiss/faiss/faiss/svs/IndexSVSVamana.cpp:162: Improper configuration for search buffer! Effective size (20) cannot be less than capacity (10). (line 86 in /workspace/bindings/cpp/build_cpp_bindings/_deps/svs-src/include/svs/index/vamana/search_buffer.h)

O_o

[aguerreb]

Also, it would be nice to have an instruction on how to install SVS. Or whether a user needs to do anything about it, package, distrubute, etc. I see that currently it is downloaded locally if you build the Faiss from the source. But the directory with a built faiss library (say, {FAISS_SRC}/build/faiss with the libfaiss_avx512_spr.so shared library file inside) does not contain any SVS-related files.

A quick how-to instruction would be appreciated by users, I suppose.

Totally agree, we'll work on that.

[aguerreb]

terminate called after throwing an instance of 'faiss::FaissException'
what(): Error in virtual void faiss::IndexSVSVamana::search(faiss::idx_t, const float*, faiss::idx_t, float*, faiss::idx_t*, const faiss::SearchParameters*) const at /home/ubuntu/faiss/faiss/faiss/svs/IndexSVSVamana.cpp:162: Improper configuration for search buffer! Effective size (20) cannot be less than capacity (10). (line 86 in /workspace/bindings/cpp/build_cpp_bindings/_deps/svs-src/include/svs/index/vamana/search_buffer.h)

O_o

The error message is incorrect, it should be "Effective size (20) cannot be larger than capacity (10)". The capacity parameter sets the size of the search buffer (that keeps track of the neighbors visited so far) and that cannot be smaller than the search window size. We'll fix that!