[cortesoft]

While I share some of the feelings about 'not understanding what is actually happening under the hood', I can't help but think about how this feeling is the exact same response that programmers had when compilers were invented:

https://vivekhaldar.com/articles/when-compilers-were-the--ai...

We are completely comfortable now letting the compilers do their thing, and never seem to worry that we "don't know what is actually happening under the hood".

I am not saying these situations are exactly analogous, but I am saying that I don't think we can know yet if this will be one of those things that we stop worrying about or it will be a serious concern for a while.

[msteffen]

I think about this a lot, though one paragraph from that article:

> Many assembly programmers were accustomed to having intimate control over memory and CPU instructions. Surrendering this control to a compiler felt risky. There was a sentiment of, if I don’t code it down to the metal, how can I trust what’s happening? In some cases, this was about efficiency. In other cases, it was about debuggability and understanding programming behavior. However, as compilers matured, they began providing diagnostic output and listings that actually improved understanding.

I would 100% use LLMs more and more aggressively if they were more transparent. All my reservations come from times when I prompt “change this one thing” and it rewrites my db schema for some reason, or adds a comment that is actively wrong in several ways. I also think I have a decent working understanding of the assembly my code compiles to, and do occasionally use https://godbolt.org/. Of course, I didn’t start out that way, but I also don’t really have any objections to teenagers vibe-coding games, I just think at some point you have to look under the hood if you’re serious.

[cortesoft]

> I would 100% use LLMs more and more aggressively if they were more transparent. All my reservations come from times when I prompt “change this one thing” and it rewrites my db schema for some reason, or adds a comment that is actively wrong in several ways.

Isn't that what git is for, though? Just have your LLM work in a branch, and then you will have a clear record of all the changes it made when you review the pull request.

[ManuelKiessling]

(I‘m saying this as someone who uses AI for coding a lot and mostly love it) Yeah, but is that really the same? Compilers work deterministically — if it works once, it will work always. LLMs are a different story for now.

[betenoire]

Said another way, compilers are a translation of existing formal code. Compilers don't add features, they don't create algorithms (unrolling, etc., notwithstanding), they are another expression of the same encoded solution.

LLMs are nothing like that

[cortesoft]

LLMs are just translating text into output, too, and are running on deterministic computers like every other bit of code we run. They aren't magic.

It is just the scope that makes it appear non-deterministic to a human looking at it, and it is large enough to be impossible for a human to follow the entire deterministic chain, but that doesn't mean it isn't in the end a function that translates input data into output data in a deterministic way.

[Izkata]

> is just the scope that makes it appear non-deterministic to a human looking at it

They are non-deterministic due to both the GPUs they run on and some additional randomness intentionally added through the "temperature" parameter.

[betenoire]

just text !== syntactically correct code that solves a defined problem

There is a world of difference between translation and generation. It's even in the name: generative AI. I didn't say anything about magic.

[cortesoft]

LLMs are deterministic, too. I know there is randomness in the choosing tokens, but that randomness is derived from a random seed that can be repeated.

[Supermancho]

Only if the seed is known. Determinism is often predicated on perfect information. Many programs do not have that. Their operations cannot be reproduced practically. The difference between saying deterministic and non-deterministic is contextual based on if you are concerned with theory or practicality.

[lelanthran]

If I understand your argument, you're saying that models can be deterministic, right?

Care to point to any that are set up to be deterministic?

Did you ever stop to think about why no one can get any use out of a model with temp set to zero?

[mrob]

llama.cpp is deterministic when run with a specified PRNG seed, at least when running on CPU without caching. This is true regardless of temperature. But when people say "non-deterministic", they really mean something closer to "chaotic", i.e. the output can vary greatly with small changes to input, and there is no reliable way to predict when this will happen without running the full calculation. This is very different behavior from traditional compilers.

[cortesoft]

No, LLMs ARE deterministic, just like all computer programs are.

I get why that is in practice different then the manner in which compilers are deterministic, but my point is the difference isnt because of determinism.

[betenoire]

I think you are misunderstanding the term "deterministic". Running on deterministic hardware does not mean an algorithm is deterministic.

Create a program that reads from /dev/random (not urandom). It's not determistic.

[cortesoft]

Fair, although you can absolutely use local LLMs in a deterministic way (by using fixed seeds for the random number generation), and my point is that even if you did that with your LLM, it wouldn't change the feeling someone has about not being able to reason out what was happening.

In other words, it isn't the random number part of LLMs that make them seem like a black box and unpredictable, but rather the complexity of the underlying model. Even if you ran it in a deterministic way, I don't think people would suddenly feel more confident about the outputted code.

[gpderetta]

LLMs are deterministic[1], but the only way to determine the output is to empirically run them. With compilers, both the implementor and a power user understand the specific code transformations they are capable of, so you can predict their output with good accuracy. I.e. LLMs are probably chaotic systems.

edit: there might be a future where we develop robopsychology enough to understand LLM more than black boxes, we we are not there yet.

[1] Aside from injected randomness and parallel scheduling artifacts.

[nextaccountic]

The difference is that compilers are supposed to be deterministic and low level inclined people often investigate compiler bugs (specially performance bugs) and can pinpoint to some deterministic code that triggered it. Fix the underlying code and it stops misbehaving with high assurance

A non deterministic compiler is probably defective and in any case much less useful

[mathieudombrock]

A major difference is that _someone_ knew what was going on (compiler devs).

[cortesoft]

That is an interesting difference, I agree.

Although, while the compiler devs might know what was going on in the compiler, they wouldn't know what the compiler was doing with that particular bit of code that the FORTRAN developer was writing. They couldn't possibly foresee every possible code path that a developer might traverse with the code they wrote. In some ways, you could say LLMs are like that, too; the LLM developers know how the LLM code works, but they don't know the end result with all the training data and what it will do based on that.

In addition, to the end developer writing FORTRAN it was a black box either way. Sure, someone else knows how the compiler works, but not the developer.

[lelanthran]

I think you have an incorrect mental model of how LLMs work.

There's plenty of resources online to rectify that, though.

[cortesoft]

I think you may be misreading my comment, then, because I know how LLMs work. Which part of my comment do you think shows that I don’t?

[gpderetta]

maybe you have a wrong mental model on how compiler works then. I'm not a compiler developer, but usually I have a pretty good idea on what code gcc will generate for my C++: it is far from a black box.

Also compilers usually compose well: you can test snippets of code in isolation and the generated code it will have at least some relation to whatever asm would be generated when the snippet is embedded in a larger code base (even under inter-procedural optimizations or LTO, you can predict and often control how it will affect the generated code).

[lelanthran]

> I think you may be misreading my comment, then, because I know how LLMs work. Which part of my comment do you think shows that I don’t?

Your repeated comparisons to compilers in the context of determinism display a lack of understanding of compilers, LLMs and determinism. That's quite an impressive trifecta!

Unless you are expecting the definition of determinism to include this: https://news.ycombinator.com/item?id=47911524 ... you can't really be surprised when people call out your definition of determinism as completely and utterly wrong.

A compiler is a function `f(x) -> y`. For given x, a specific compiler will in actual use always result in the specified y. There is no currently used LLM that I am aware of that is able to do that for any non-trivial input.

An LLM is a function `f(x) -> P(y)`. For a given x, you get a probability of y as the output.

If you are still unable to acknowledge that there is difference between `f(x) -> y` and `f(x) -> P(y)`, then there really isn't any reason to continue this thread.

[mnkypete]

Except that compilers are (at least to a large degree) deterministic. It's complexity that you don't need to worry about. You don't need to review the generated assembly. You absolutely need to review AI generated code.

[seamossfet]

People often use "non-deterministic" to imply "unreliable" when that's not always the case. You can have an LLM that's clearly non-deterministic but also reliable for tasks.

[cortesoft]

At the end of the day, LLMs are also deterministic. They are running on computers just like all software, and if you have all the same data and random seeds, and you give the same prompt to the same LLM, you will get back the exact same response.

[Supermancho]

> you give the same prompt to the same LLM, you will get back the exact same response.

Demonstrably incorrect. This is because the model selection, among other data, is not fixed for (I would say most) LLMs. They are constantly changing. I think you meant something more like an LLM with a fixed configuration. Maybe additional constraints, depending on the specific implementation.

[cortesoft]

Yes, by 'same LLM', I mean literally the same model with the same random seeds. You are correct, the big LLMs from providers like Anthropic and OpenAI do not meet this definition.