[kristiandupont]

It's obviously an analogy, but it seems pretty fitting to me? What would you call it?

[TillE]

Making errors, generating nonsense, being wrong. It's a catchy term but it's not accurate in any meaningful way.

[seadan83]

Hallucinating has the implication of being wrong. The word further adds the context of being elaborately wrong. That feels pretty accurate to describe an AI going into detail when it is wrong.