[helen___keller]

I don’t think that’s necessarily a problem. Technically every time you update gcc your compiler might produce a different underlying program for your source code.

The bigger problem is that LLMs are slow and expensive. Even in the future after many improvements, it makes more sense to have an LLM write a program once ever, rather than write a program on every compile or every execution.

[ch4s3]

> Technically every time you update gcc your compiler might produce a different underlying program for your source code.

Individual versions are deterministic though. Two identical prompts to a LLM at the same time can give drastically different results, because the the responses are probabilistic. You can't assemble complicated systems that way and expect them to behave consistently.

[helen___keller]

The issue you’re describing is more to do with correctness and performance (two critical elements of a good compiler), not nondeterminism.

If a natural language compiler can output correct performant code, nondeterminism shouldn’t matter.

For example, take a script that randomly invokes either gcc or clang, maybe randomly sets the optimization level. Multiple invocations will output vastly differently, but we can be confident the output is correct and to some degree performant.

[ch4s3]

No my point is that nondeterminism effects correctness. A random script that invokes different compilers is a contrived example, no one would ever build a system that way it is totally undesirable. Moreover, I'm not sure how we could determine the correctness of a system generated by an LLM without auditing the output and certifying each run. Who would ever want to work that way? This just creates problems that don't need to exist.

[helen___keller]

> Who would ever want to work that way? This just creates problems that don't need to exist.

That depends on what you want.

In the first place, the problem of compiling a natural language spec to code is obviously somewhere from undefined to Turing complete (depending on formulation). But if the compiler usually outputs some application with most of what the spec required, this compiler would be intensely useful for e.g. rapid prototyping.

Then the question is whether we can make an LLM based app that compiles natural language and gets you most of the way to the prototype you were building (or even better - asks clarifying questions to help refine your spec).

This isn’t that far fetched with current technology.

[ch4s3]

I can totally see the case for prototyping, it makes some sense. I just think that by the time you are specifying something so clearly that the results are correct you may well be practically programming in a super high level dynamic language.