|
|
|
|
|
|
by bambax
489 days ago
|
|
|
The problem is what one means by "works". Is it just that it runs without triggering exceptions here and there? One has to know, and understand, what the code is supposed to be doing, to evaluate it. Or use tests. But LLMs love to lie so they can't be trusted to write the tests, or even to report how the code they wrote passed the tests. In my experience the way to use LLMs for coding is exactly the opposite: the user should already have very good knowledge of the problem domain as well as the language used, and just needs to have a conversation with someone on how to approach a specific implementation detail (or help with an obscure syntax quirk). Then LLMs can be very useful. But having them directly output code for things one doesn't know, in a language one doesn't know either, hoping they will magically solve the problem by iterating in "closed loops", will result in chaos. |
|
|
The thing everyone thinks about with LLM codegen is hallucination. The biggest problem for LLMs with hallucination is that there are no guardrails; it can just say whatever. But an execution environment provides a ground truth: code works or it doesn't, a handler path generates an exception or it doesn't, a lint rule either compiles and generates workable output or it doesn't.