Benchmark

ANN Search Benchmark

codelibs/search-ann-benchmark evaluates the performance of various Approximate Nearest Neighbor (ANN) algorithms, comparing both response time and accuracy metrics. This provides a comprehensive comparison of ANN-enabled systems for handling high-dimensional data spaces.

Overview

The tests focus on two main metrics:

  • QTime(msec): The time taken to respond to a search query, calculated as the average time for 10,000 queries.
  • Precision@K: The accuracy of the search results for K=10 and K=100.

The tables also include columns labeled "Top 10" and "Top 100," which indicate the number of results retrieved:

  • Top 10: Retrieves the top 10 results.
  • Top 100: Retrieves the top 100 results.

Results

The tests involve searching through a dataset of 100,000 vectors with 768 dimensions, yielding the following results:

ANN Search (Vector Only)

Product Top 10 Top 100 Test Date
QTime Precision@10 QTime Precision@100
chroma 0.5.7 5.2482 0.99225 8.0238 0.95742 2024-09-26
elasticsearch 8.15.2 2.6260 0.99730 9.3898 0.99063 2024-09-28
elasticsearch 8.15.2 (int8) 1.9996 0.95861 8.8340 0.96451 2024-09-28
elasticsearch 8.15.2 (int4) 5.7716 0.81929 12.4838 0.84643 2024-09-28
milvus 2.4.12 3.9257 0.92656 4.4710 0.96553 2024-10-12
opensearch 2.17.0 1.8737 0.89207 10.5771 0.99306 2024-09-22
opensearch 2.17.0 (faiss) 4.5027 0.99974 10.1445 0.99753 2024-09-22
pgvector 0.7.4-pg16 20.0423 0.99623 20.3250 0.97684 2024-09-14
qdrant 1.12.0 2.1949 0.99976 2.2760 0.99770 2024-10-09
qdrant 1.12.0 (int8) 1.0798 0.92305 1.2451 0.94046 2024-10-09
vespa 8.416.42 1.8132 0.99110 2.1328 0.95153 2024-10-07
weaviate 1.26.3 5.8845 0.99271 6.8662 0.95675 2024-09-01

ANN Search with Keyword Filtering

Product Top 10 Top 100 Test Date
QTime Precision@10 QTime Precision@100
chroma 0.5.7 - - - - 2024-09-26
elasticsearch 8.15.2 2.7646 0.99899 9.8507 0.99925 2024-09-28
elasticsearch 8.15.2 (int8) 1.6522 0.95606 8.8173 0.96996 2024-09-28
elasticsearch 8.15.2 (int4) 5.0625 0.81008 12.1456 0.85704 2024-09-28
milvus 2.4.12 4.4185 0.92319 5.0487 0.92495 2024-10-12
opensearch 2.17.0 3.2763 0.99506 9.2618 0.99994 2024-09-22
opensearch 2.17.0 (faiss) 2.0240 1.00000 7.4407 0.99999 2024-09-22
pgvector 0.7.4-pg16 20.7157 0.34083 20.7150 0.05645 2024-09-14
qdrant 1.12.0 0.9420 0.99988 1.0910 0.99954 2024-10-09
qdrant 1.12.0 (int8) 0.6881 0.92449 0.8186 0.94889 2024-10-09
vespa 8.416.42 3.9992 0.99979 4.2051 0.99367 2024-10-07
weaviate 1.26.3 6.6821 0.99991 7.8699 0.99988 2024-09-01

The tests are run on GitHub Actions, and the results are collected and summarized in tables. These benchmarks provide basic reference values, allowing users to evaluate and select an appropriate system based on their specific requirements. Be sure to test and verify the chosen solution's performance in your particular context before deployment.