[xantronix]

The thing with LLMs is that it is very, very easy to adjust the weights across the entire model to sway responses one way or another. Previously, in the hypothetical case one wanted to rewrite history, it would be a much more involved endeavour of curation; fabrication of original sources would be difficult to do at scale. But now it's trivial for a provider to inject a preamble to the prompt to not only hide results that do not fit the narrative of those legislating in the model providers' favour, but to distort the results.

Obviously none of that is happening in the current moment, and I grant that cake recipes would be low stakes, but I would rather take the tradeoff of trawling through a little bit of slop to get that same information than acclimate myself to a workflow that could be abused by providers in more high-stakes situations down the line.

But that's just me, and I realise this is not a particularly popular take, but it should nonetheless be illustrative for why "just ask the LLM" might not be the best of ideas long term.