Hacker News new | ask | show | jobs
by orbital-decay 14 days ago
I think you want the impossible, because a) input semantics are non-formal and ambiguous/subjective by definition, and b) the model suffers from the curse of its knowledge being vastly wider than yours and doesn't have enough context to converge on exactly what you want in the huge space of possibilities presented by even the most constraining but still informal inputs.

If you limit your requirement to the difference between your and model's interpretations being small enough, that's probably doable. Which is realistically what most people want, and most good coding models already have, more or less (with caveats that still need to be addressed, of course). But a hard guarantee of output staying unchanged with different inputs is not possible to give (regardless of whether you think they're unambiguous) due to the nature of intelligence, human or machine.

1 comments

I'm not asking for LLM tools to be similar to compilers, or saying that they can't be useful if they aren't. I know rather well that the two are different, and that's the point.

Because LLMs aren't deterministic in terms of producing semantically correct output, that just means they aren't similar to compilers. That means you probably can't just start blindly trusting that their output matches the input and thus ignore understanding the code, as most people mostly can with compilers.

I think that's what people mean with "determinism" when they compare LLMs to compilers, or in response to other people suggesting LLMs are no different than compilers.