[jnnnthnn]

Thanks! I tried a few other approaches and found the LLM results were overall better (latency and cost aside). Maybe that should be an option made available to users though...

[isoprophlex]

Cohere has a very cheap, fast and effective reranking API!

https://cohere.com/rerank

[bardiapour]

i think not, better results >>> better latency + cost

[robrenaud]

Maybe a combined approach beats either? Let some non-LLM reranker quickly spit out two results, and fill in the rest with the LLM.