[dartos]

I’ve heard this comparison between C compilers and LLMs and I don’t think it’s similar.

there’s a direct relationship to C code and the resulting assembly.

The C language was specifically designed so that language constructs were converted into the intended assembly anyway.

C didn’t obfuscate the problem of writing code, just allowed us to operate at a higher, but still sound, level.

LLMs are not an abstraction like C. You aren’t going through a sound, tractable process going from natural language to code.

You’re going through a black box.

It’s no longer a new abstraction on top of a fundamental idea.

There is no understanding of anything being developed. Prompts aren’t even deterministic wrt the output of the LLM, so even how to prompt a given model isn’t something that can be understood or totally predicted. It’s just like SEO hacks.

[hnfong]

While C compilers are technically not "black boxes", if you looked at lawyerly discussions about UB I don't think you can in full conscience say that `gcc -O2` is much better than a LLM.

[dartos]

They’re not black boxes at all. The code is all there and is deterministic.

Sorry, but the peculiarities of one optimization level on one implementation of C do not affect my point at all.

[hnfong]

I already said they are technically not black boxes. We are not in disagreement here.

The fact that code is all there and deterministic isn't sufficient. People use newer versions of compilers and you can't predict what the people who write compilers will do (people are black boxes). The compilers may do something you don't expect but still conform to the spec.

The point is that there is no one implementation of C. And the same is true for any other language too (to a lesser extent).

Unless you're coding against very specific versions of OS and compiler and runtime environment, you don't have code to inspect. You are not inspecting all the underlying dependencies when you write your code. You just assume they work.

It doesn't matter whether they are black boxes if you don't routinely inspect the boxes. Perhaps you do, but for most people they don't. So it doesn't matter as much.

[dartos]

> I already said they are technically not black boxes. We are not in disagreement here.

I’m saying they are not black boxes at all. Not just “technically”

> The point is that there is no one implementation of C. And the same is true for any other language too (to a lesser extent).

That doesn’t matter for my argument.

> It doesn't matter whether they are black boxes if you don't routinely inspect the boxes. Perhaps you do, but for most people they don't. So it doesn't matter as much.

It does matter. That’s my whole point.

For two main reasons:

First, you have the option to understand how a compiler translates C to assembly. You do NOT have that option with LLMs

With a compiler, you are abstracting the underlying machine code, but you’re not hiding it entirely.

Secondly, with an LLM there’s no tractable connection between your prompt and your code. That’s a huge issue when it comes to understanding the resulting program.

Imagine a world where LLMs turn raw language into assembly (or, more realistically, LLVM IR)

How would you reason about your resulting program?

You could give the same prompt to the same LLM and get wildly different resulting code, since LLMs are nondeterministic.

How would you debug? How would you ensure that small changes to your LLM prompt doesn’t change parts of your resulting code that worked well?

How would you even understand what your program is doing?

You couldn’t do any of those things, because there’s no deterministic relationship between the prompt and the resulting program.

[hnfong]

LLMs can be deterministic if you set the temp to zero.

It's not feasibly debuggable though, I'll give you that.

On this point it's not really different in principle from having a non-OSS compiler. Using OSS compilers are great because you can look at the source and trace problems and fix them, but people also use closed source compilers in their work too.

[dartos]

> LLMs can be deterministic if you set the temp to zero.

At which point, you the major benefit statistical models like of LLMs…

A lot of performance is lost if you fix the sampling step like that as well.

That’s why nobody does it…