[emil_sorensen]

Exactly! It's very much garbage-in-garbage-out.

One surprising (to me at least) benefit of hooking up an LLM to your docs is that it is actually a really useful way to find gaps in your docs. For example, when an LLM cannot answer a user question, there's a good chance it's because the answer is not documented anywhere.

[thfuran]

What about when it answers with confabulation?

[mirekrusin]

Confabulation should not be stable. Ie. you can generate answer let's say 3 times with different seed / non-zero temp and if it arrives at different answers you can categorize it as confabulation - which likely means, again, that the answer is not present and humans will likely also have different interpretations.

[CamperBob2]

I guess then it becomes a threat to Google's business model.