[ryandrake]

It's a defect from the point of view of user expectations. When Intel's floating point bug was in the news, I remember a small number of people claiming it was not a defect because the chip was just doing what it was designed to do: Yea, it was designed in such a way that it could produce incorrect results. In other words a bug!

I'm sure AI companies will get very good at explaining away these defects with various forms of "aCkShUaLlY" but when your marketing materials say you made a box that takes a prompt and answers it, and it answers incorrectly, what else is it than a defect?

[krapp]

The problem, in that case, exists between the keyboard and chair.

Floating point math is inherently inaccurate, and no programmer using it would expect perfect precision and call it a defect not to get it. You have to understand how floating point works and take that inaccuracy into account. As a result there are some applications for which using floats is simply a bad idea. No one sane is doing real money calculations with floats.

The same goes for LLMs. Hallucination is fundamental to the model. We're going to have to realize that there are many tasks for which AI simply isn't well suited. And we're going to have to get over this persistent delusion that humans are categorically worse than AI at everything. A paralegal doing research would probably not simply fabricate cases and cites whole cloth. That's not how most humans work. Humans are capable of knowing when they don't know something, AI is not.

But we've decided, for whatever reason, that AI is perfectly trustworthy. That's going to keep biting us in the ass until we learn.

[ryandrake]

I don't think your memory of the Intel bug[1] is correct. It had nothing to do with the inherent precision problems of floating point representation.

EDIT: Another way to put it: If I sold a calculator that claimed to do math, but in the fine print I said "Actually, it just makes up answers by some means we don't fully understand, and somehow most of the time it comes up with the right answer." That doesn't mean that incorrect answers are suddenly not defects.

1: https://en.wikipedia.org/wiki/Pentium_FDIV_bug

[dmoy]

Yea it fundamentally changes the error bounding calculation. We explicitly learned about this in EE where they covered 754 and other similar things.

So there's a lot of rigor on containing exactly how big your errors can be in floating point calculations, and that old intel bug made it (rarely) invalid.