Hacker News new | ask | show | jobs
by seanmcdirmid 8 days ago
This. The math of coincident errors is why we can get reliable AI, but it really is a matter of how the errors manifest and how they are detected.

I’ve written a bit about this and my approach is based on clean room and N-versions.
1 comments
camgunz 8 days ago
Reliable AI is an oxymoron and deeply misleading. Even if you can stack models the efficiency tanks. Also, just looking at the Wikipedia page (https://en.wikipedia.org/wiki/N-version_programming#:~:text=...), don't LLMs already have all these problems?
link
seanmcdirmid 7 days ago
You aren’t thinking this through. You were just told that this wasn’t stacking but you keep insisting it is. N-versions is explicitly not stacking, and the same solutions that solve the Knight and Leveson issue for humans solves it for LLMs as well (they can be wrong about the same part of the specification, but will very unlikely be wrong about it in the same way, so you try to measure scalar rather than Boolean outcomes).
link
camgunz 7 days ago
> You aren’t thinking this through.

No need to be so polite; I might just be very stupid.

> You were just told that this wasn’t stacking but you keep insisting it is.

Having an LLM generate something and then asking another LLM to generate that something is asking for 2 hallucinations, not 1. This doesn't go away the more LLMs you throw in there. The only reason this works at all for things like ChatGPT is their prompts are super different (and the editor LLM is way cheaper, of course).

And you still have to check for hallucinations (before you say throw another hallucinating LLM at it know I will make you pay infinite forum shame points if you do).

> the same solutions that solve the Knight and Leveson issue for humans solves it for LLMs as well

No, LLMs are plagued by the same problems: https://arxiv.org/abs/2606.20158. After all what are you gonna do, say "build me a SQLite, but definitely not like [other model X] would"? The solution is a very unambiguous spec, but that asymptotically approaches code soooooo....

Finally, I think you're hand-waving NVX. A program that can digest any spec and measure how well any program meets it is pretty wild. Good luck with that one.

link