Hacker News new | ask | show | jobs
by aprilthird2021 766 days ago
It's very commonly understood by those of us who actually produce and consume AI research that "knowing how" LLMs (and Neural Nets for that matter) work doesn't mean knowing how to build one. It means mathematically proving and understanding "how" the steps we put the LLM through when training are able to produce the output we get when testing.

We know how to build it. We don't understand how it's producing the output it does based off what we give it
2 comments
kybernetikos 766 days ago
This still doesn't make sense to me. As far as I'm concerned the gold standard of understanding something is being able to construct a program that replicates it, which is exactly what we can do with LLMs.

We know exactly how llms work (relatively simple maths), and to a large extent even why they work (backpropagation updates weights to more closely approximate the desired function). There are open questions relating to LLMs of course - we don't understand what the space of potential LLM-like things looks like and how the features in that space relate to subjective performance (although note that transformers were designed based on a theory that they would perform better, not just randomly generated or inspired by the muse). We also don't know to what extent the output of LLMs can be approximated by simpler symbolic systems, or how to extract such systems from LLMs when they do exist. Those are really interesting questions, but they're not questions about 'how LLMs work'.

I dislike the 'LLMs are magic' framing that seems to be taking over the world. Nobody thinks that Taylor expansion is magical, but LLMs are doing the same sort of thing - approximating a function through a bunch of weights on a bunch of simpler functions. Just because the function we're approximating (intelligent output) is not known in advance (but can be sampled), and multi-dimensional does not fundamentally change how mysterious the process is.

link
aprilthird2021 760 days ago
> the gold standard of understanding something is being able to construct a program that replicates it

Cloning animals or even humans did not automatically make us understand how brains work. In fact, these were quite unrelated endeavors.

> I dislike the 'LLMs are magic' framing that seems to be taking over the world

Don't take that out on me. That's not what I'm saying. I'm saying there is a lack of determinism (mathematically provable, per se) in our current understanding of all AI (LLM included). There are many attempts to solve this problem. I've sat in on seminars about it myself. So far, we're not there yet

link
kybernetikos 760 days ago
> Cloning animals or even humans did not automatically make us understand how brains work. In fact, these were quite unrelated endeavors.

I agree. It's not copying that I'm saying is understanding, it's modelling.

> I'm saying there is a lack of determinism (mathematically provable, per se)

What do you mean by a lack of determinism in this case?

link
OrderlyTiamat 766 days ago
> As far as I'm concerned the gold standard of understanding something is being able to construct a program that replicates it which is exactly what we can do with LLMs.

But we actually can't! We can build a program that can build a program that is the LLM, which is not the same! I'd argue that you're right insofar as training is concerned. We understand training very well. But the actual model, how it operates, what it actually knows, we don't know how to build that, we don't know what weights to put where.

link
Izkata 765 days ago
We actually have another example of this as well:

Malbolge is an esoteric programming language designed to be impossible to use. The first program written in it wasn't written by a human, it was written by another program.

But since then, with that working program to learn from, people have figured out how to write programs in malbolge: https://lutter.cc/malbolge/tutorial/cat.html

link
AstralStorm 766 days ago
It's a bit similar to the problem of neuroscience. We understand how a single neuron works pretty well, or even a small count of them. Even a few subsystems like balance or lower level vision. A bit on muscle control and endocrine system.

We do not understand language, grammar, music, only partly emotions, or especially sentience and consciousness. Further, we don't understand how the disparate systems are integrated together.

link
mvkel 766 days ago
Great comparison.

The comparison itself is pretty telling; that the brain and AI work so similarly in specific ways

link