|
|
|
|
|
|
by andai
889 days ago
|
|
|
I tried it a few times but was mostly disappointed with the results. I built a superior solution (for my usage anyway) in like 50 lines of Python. Just uses duckduckgo for search, extracts text from top 8 results, shoves into gpt-3. (I think I got a massive cost reduction by filtering paragraphs with embedding API, but that's just an optimization.) Phind is better than what I built, though. I don't actually use it that much (I'm not sure why—habit? Impatience?), but every time I do I'm surprised with the accuracy. I think a big part of it is which pages to select for sources. Google Bard failed hard here in my testing, gave nonsense answers because its sources were AI generated SEO spam! |
|
|
It seems that the main problem right now is that they only look at the first few results. Basically they all do this:
1. Ask model for a search query.
2. Run search and feed the results into the model.
3. Ask model for an answer.
The problem is that for very specific questions the search results won't contain the required information. So it has to say something irrelevant or hallucinate. Especially for negative queries like "a foo library that doesn't depend on std" or similar it really struggles because it can't effectively filter out libraries that fail the requirements and can't keep searching until it finds one.
Basically they can be fast if a decent query can have the answer within the first page of results. But fail otherwise. In many cases you are better off just reading through search results on your own.