Benchmark

codelibs/search-ann-benchmark evaluates the performance of various Approximate Nearest Neighbor (ANN) algorithms, comparing both response time and accuracy metrics. This provides a comprehensive comparison of ANN-enabled systems for handling high-dimensional data spaces.

Overview

The tests focus on two main metrics:

• QTime(msec): The time taken to respond to a search query, calculated as the average time for 10,000 queries.
• Precision@K: The accuracy of the search results for K=10 and K=100.

The tables also include columns labeled "Top 10" and "Top 100," which indicate the number of results retrieved:

• Top 10: Retrieves the top 10 results.
• Top 100: Retrieves the top 100 results.

Results

The tests involve searching through a dataset of 100,000 vectors with 768 dimensions, yielding the following results:

ANN Search (Vector Only)

Product	Top 10		Top 100		Test Date
	QTime	Precision@10	QTime	Precision@100
chroma 1.5.7	3.1464	0.99227	4.8622	0.95803	2026-05-16
elasticsearch 9.5.1	0.8469	0.99646	1.0580	0.98229	2026-08-20
elasticsearch 9.5.1 (int8)	0.3692	0.98279	0.9908	0.97761	2026-08-20
elasticsearch 9.5.1 (int4)	0.1540	0.91684	0.8054	0.93052	2026-08-20
elasticsearch 9.5.1 (bbq)	0.4271	0.92991	1.0750	0.96687	2026-08-20
elasticsearch 9.5.1 (bbq_disk)	0.0511	0.94242	0.9875	0.96784	2026-08-20
elasticsearch 9.5.1 (bbq_disk_4bit)	0.3622	0.92312	1.0290	0.93655	2026-08-20
elasticsearch 9.5.1 (bbq_disk_2bit)	0.2538	0.93397	1.0142	0.95794	2026-08-20
milvus 2.6.14	3.5668	0.93248	4.0373	0.96619	2026-05-16
opensearch 3.8.0	1.9629	0.99680	6.8963	0.99012	2026-08-20
opensearch 3.8.0 (faiss)	5.4446	0.99869	10.8220	0.99676	2026-08-20
pgvector 0.8.2-pg18	5.6219	0.99612	7.1959	0.97693	2026-05-16
qdrant 1.19.0	1.5270	0.99609	1.9637	0.97394	2026-08-21
qdrant 1.19.0 (int8)	0.7940	0.93050	1.1937	0.93550	2026-08-21
vespa 8.738.17	1.7410	0.99040	1.8906	0.95159	2026-08-22
weaviate 1.36.9	5.1780	0.99214	6.2161	0.95663	2026-05-16
redisstack 7.4.0-v8	0.9071	0.99185	2.3524	0.95648	2026-05-16
clickhouse 26.2	5.9794	0.95506	6.8066	0.91936	2026-05-16
lancedb 0.29.2	43.2166	0.99907	72.7166	0.99918	2026-05-16
vald v1.7.17	0.8809	0.84717	4.1030	0.94217	2026-05-16

ANN Search with Keyword Filtering

Product	Top 10		Top 100		Test Date
	QTime	Precision@10	QTime	Precision@100
chroma 1.5.7	45.0852	0.99866	46.8925	0.99139	2026-05-16
elasticsearch 9.5.1	0.5065	0.99013	0.9508	0.97997	2026-08-20
elasticsearch 9.5.1 (int8)	0.3684	0.97890	1.0263	0.97636	2026-08-20
elasticsearch 9.5.1 (int4)	0.1731	0.91366	0.7066	0.93642	2026-08-20
elasticsearch 9.5.1 (bbq)	0.4701	0.93207	1.2252	0.98144	2026-08-20
elasticsearch 9.5.1 (bbq_disk)	0.0433	0.93648	0.8344	0.96501	2026-08-20
elasticsearch 9.5.1 (bbq_disk_4bit)	0.3875	0.90899	1.1776	0.91892	2026-08-20
elasticsearch 9.5.1 (bbq_disk_2bit)	0.2279	0.92121	1.0536	0.94193	2026-08-20
milvus 2.6.14	3.9386	0.92696	4.8288	0.92512	2026-05-16
opensearch 3.8.0	2.1248	0.99899	8.6568	0.99924	2026-08-20
opensearch 3.8.0 (faiss)	3.5013	0.99725	8.3575	0.99450	2026-08-20
pgvector 0.8.2-pg18	40.7029	1.00000	41.4975	1.00000	2026-05-16
qdrant 1.19.0	0.8173	0.99822	1.0992	0.99250	2026-08-21
qdrant 1.19.0 (int8)	0.4843	0.93223	0.7756	0.95259	2026-08-21
vespa 8.738.17	1.7463	0.95832	2.0663	0.95356	2026-08-22
weaviate 1.36.9	6.0454	0.99891	7.1748	0.99919	2026-05-16
redisstack 7.4.0-v8	2.2221	0.99502	4.3616	0.99929	2026-05-16
clickhouse 26.2	8.6085	0.33553	12.9999	0.40527	2026-05-16
lancedb 0.29.2	41.8065	0.99899	72.2861	0.99929	2026-05-16
vald v1.7.17	4.1827	0.33377	28.9823	0.40107	2026-05-16

The tests are run on GitHub Actions, and the results are collected and summarized in tables. These benchmarks provide basic reference values, allowing users to evaluate and select an appropriate system based on their specific requirements. Be sure to test and verify the chosen solution's performance in your particular context before deployment.