[nostrebored]

Oh I agree with you, it's just that I don't think LLMs are either. If you think of LLM knowledge, especially in scientific/engineering fields, as a lossy representation of the density of ideas, then you'd expect to see some weird behavior. I'm sure there is some sort of a temporal discounting and people thinking about this, but a naive NLL or Reverse-KL on medical literature would engrain some weird, wrong ideas.

[StefanBatory]

I brought down prod system because I was too stupid and relied down on LLM advice, even double checking it.

I would not trust my health to LLMs, that is true.