[alanfranz]

> How good are you at writing assembly?

This is a false equivalence.

A compiler is a predictable, testable, deterministic piece of software.

An LLM is not.

Sure, all abstractions leak; so, at some point in time, for some reason, you may need to check its compiled code ( cough cough gcc 2.96 ). But, if today your code compiles properly, it will properly compile tomorrow as well.

[js8]

LLMs can be deterministic as well - same prompt on the same model produces the same input. On the other hand, compilers can be quite undeterministic - you get a new version of compiler, or change compiler options (turn on optimizations) - you might get a very different binary. And JIT compilers (and GC languages) even less deterministic, their compilation can depend on the nature of the inputs.

But I think, in the analogy compiler ~ LLM, the issue is more of a trust than determinism. It took decades to assembler programmers to trust compilers enough not to write code in assembler. The similar will happen with AI - some will embrace it sooner than others.

[skydhash]

> LLMs can be deterministic as well - same prompt on the same model produces the same input

> compilers can be quite undeterministic - you get a new version of compiler, or change compiler options (turn on optimizations)

That’s a whole other level pf bad faith argument right here. Flags and options are input too.

> It took decades to assembler programmers to trust compilers enough not to write code in assembler.

You do realize that Cobol, Algol, and Lisp are very old, and they were not assembly. And that Unix were written in C shortly after the language was created.

[js8]

> That’s a whole other level pf bad faith argument right here.

Not sure where you see the bad faith argument. (Btw I mean "same output", not "same input", it was a typo.)

Take for example JVM. It used to be horribly bad and unpredictable, performance wise, in the 90s. Sun tried to base a desktop environment on it - it didn't work.

> You do realize that Cobol, Algol, and Lisp are very old, and they were not assembly.

Of course! But people have been hand-writing assembler until late 2000s, because compilers were simply not that good.

The same will happen with LLMs - some people will not trust it and won't use it for decades, possibly. Some have already embraced it.

[skydhash]

> Not sure where you see the bad faith argument

You proof for your argument that a compiler is undeterministic is to change the whole compiler to another version and saying it won’t produce the same output as the old one.

> But people have been hand-writing assembler until late 2000s, because compilers were simply not that good.

And we have software like Unix, enacs, ksh, awk… that’s all written in C. I strongly believe that those people who were writing assembly was optimizing stuff or dealing with constraints (like the 640kb of DOS). Just like today, you may still have to write assembly for microcontrollers or video codecs. Compilers were expensive, but people were paying for them.

[js8]

> You proof for your argument that a compiler is undeterministic is to change the whole compiler to another version and saying it won’t produce the same output as the old one.

Fair enough. What I meant though was that compilation as a process is not deterministic, because often when you recompile couple years later, you're using a different compiler. (In modern world it can be much shorter time, actually.)

> And we have software like Unix, enacs, ksh, awk… that’s all written in C.

So? IIRC, first compiler was FORTRAN, invented in 1958. OpenAI Codex, first coding LLM, came out August 2021. So we are like in a year 1963. For this comparison, we have ten more years to produce (using a coding LLM) a compiler and operating system just from the textual specification, without an intermediate formal programming language. Funny - we have actually already done that (Claude C Compiler, VibexOS).

[skydhash]

> So? IIRC, first compiler was FORTRAN, invented in 1958. OpenAI Codex, first coding LLM, came out August 2021. So we are like in a year 1963. For this comparison, we have ten more years to produce (using a coding LLM) a compiler and operating system just from the textual specification, without an intermediate formal programming language.

Nope, the timeframe would have been three years

  In 1961, the MCP was the first OS written exclusively in a high-level language (HLL).[0]

So by 2024, we should all have been able to verify that LLMs are reliable to produce a good enough product. Instead, it’s just slop everywhere, where the one producing it does not even care about its creation.

[0]: https://en.wikipedia.org/wiki/Burroughs_MCP#History