[binarymax]

The reason https://ann-benchmarks.com is so good, is that we can see a plot of recall vs latency. I can see you have some latency numbers in the leaderboard at the bottom, but it's very difficult to make a decision.

As a practitioner that works with vector databases every day, just latency is meaningless to me, because I need to know if it's fast AND accurate, and what the tradeoff is! You can't have it both ways. So it would be helpful if you showed plots showing this tradeoff, similar to ann-benchmarks.

[liliuleo93]

Thanks for your suggestion and this is a super good question. I was asked some times and please allow me quote one of my response in the repo

" With respect to recall vs Performance, your idea is indeed correct. However, several reasons have guided us to our current approach:

1. We are not solely benchmarking open-source systems; we are also focusing on cloud services. Some of these services, such as Zilliz and Pinecone, don't allow users to customize their parameters to tune the recall, aiming to simplify their usage. Consequently, creating a recall vs Performance graph is not feasible. Also this benchmark allow users to customize their parameters for systems allowing tuning to get their own result to do comparison.

2. There already exists a number of benchmarks doing what you've suggested, which target individuals with ANN search backgrounds. Our goal is to make this benchmark as straightforward as possible and to assist people who lack understanding about the inner workings of each system.

3. Concerning reproducibility, generating a recall vs QPS graph that you mentioned, would require conducting a multitude of tests to obtain enough data points, which considerably reduces reproducibility. "

the link is: https://github.com/zilliztech/VectorDBBench/issues/200#issue...