ANN Search Benchmark
codelibs/search-ann-benchmark evaluates the performance of various Approximate Nearest Neighbor (ANN) algorithms, comparing both response time and accuracy metrics. This provides a comprehensive comparison of ANN-enabled systems for handling high-dimensional data spaces.
The tests focus on two main metrics:
The tables also include columns labeled "Top 10" and "Top 100," which indicate the number of results retrieved:
The tests involve searching through a dataset of 100,000 vectors with 768 dimensions, yielding the following results:
| Product | Top 10 | Top 100 | Test Date | ||
|---|---|---|---|---|---|
| QTime | Precision@10 | QTime | Precision@100 | ||
| chroma 0.5.7 | 5.2482 | 0.99225 | 8.0238 | 0.95742 | 2024-09-26 |
| elasticsearch 8.15.2 | 2.6260 | 0.99730 | 9.3898 | 0.99063 | 2024-09-28 |
| elasticsearch 8.15.2 (int8) | 1.9996 | 0.95861 | 8.8340 | 0.96451 | 2024-09-28 |
| elasticsearch 8.15.2 (int4) | 5.7716 | 0.81929 | 12.4838 | 0.84643 | 2024-09-28 |
| milvus 2.4.12 | 3.9257 | 0.92656 | 4.4710 | 0.96553 | 2024-10-12 |
| opensearch 2.17.0 | 1.8737 | 0.89207 | 10.5771 | 0.99306 | 2024-09-22 |
| opensearch 2.17.0 (faiss) | 4.5027 | 0.99974 | 10.1445 | 0.99753 | 2024-09-22 |
| pgvector 0.7.4-pg16 | 20.0423 | 0.99623 | 20.3250 | 0.97684 | 2024-09-14 |
| qdrant 1.12.0 | 2.1949 | 0.99976 | 2.2760 | 0.99770 | 2024-10-09 |
| qdrant 1.12.0 (int8) | 1.0798 | 0.92305 | 1.2451 | 0.94046 | 2024-10-09 |
| vespa 8.416.42 | 1.8132 | 0.99110 | 2.1328 | 0.95153 | 2024-10-07 |
| weaviate 1.26.3 | 5.8845 | 0.99271 | 6.8662 | 0.95675 | 2024-09-01 |
| Product | Top 10 | Top 100 | Test Date | ||
|---|---|---|---|---|---|
| QTime | Precision@10 | QTime | Precision@100 | ||
| chroma 0.5.7 | - | - | - | - | 2024-09-26 |
| elasticsearch 8.15.2 | 2.7646 | 0.99899 | 9.8507 | 0.99925 | 2024-09-28 |
| elasticsearch 8.15.2 (int8) | 1.6522 | 0.95606 | 8.8173 | 0.96996 | 2024-09-28 |
| elasticsearch 8.15.2 (int4) | 5.0625 | 0.81008 | 12.1456 | 0.85704 | 2024-09-28 |
| milvus 2.4.12 | 4.4185 | 0.92319 | 5.0487 | 0.92495 | 2024-10-12 |
| opensearch 2.17.0 | 3.2763 | 0.99506 | 9.2618 | 0.99994 | 2024-09-22 |
| opensearch 2.17.0 (faiss) | 2.0240 | 1.00000 | 7.4407 | 0.99999 | 2024-09-22 |
| pgvector 0.7.4-pg16 | 20.7157 | 0.34083 | 20.7150 | 0.05645 | 2024-09-14 |
| qdrant 1.12.0 | 0.9420 | 0.99988 | 1.0910 | 0.99954 | 2024-10-09 |
| qdrant 1.12.0 (int8) | 0.6881 | 0.92449 | 0.8186 | 0.94889 | 2024-10-09 |
| vespa 8.416.42 | 3.9992 | 0.99979 | 4.2051 | 0.99367 | 2024-10-07 |
| weaviate 1.26.3 | 6.6821 | 0.99991 | 7.8699 | 0.99988 | 2024-09-01 |
The tests are run on GitHub Actions, and the results are collected and summarized in tables. These benchmarks provide basic reference values, allowing users to evaluate and select an appropriate system based on their specific requirements. Be sure to test and verify the chosen solution's performance in your particular context before deployment.