[defmacr0]

Well at least it seems pretty implausible to me that a machine learning model trained to reproduce human text can in principle generate something that is significantly above human text production ability. If the "pea-sized semiconductor brain" is not a surprisingly shallow problem that you can just solve by interpolating existing research, I don't really see the LLM-approach to AI being the thing that makes something like it happen.

>we do use simulated evolution as a standard technique in machine learning.

Well, not really. For large-scale AI models it's almost exclusively some form of gradient-based non-linear optimization. Genetic algorithms (which is just hill-climbing optimization with extra steps) and genetic programming (which is really cool and not well-understood) do not perform all that well in practice and I'm not aware of any notable applications.

[ben_w]

> Well at least it seems pretty implausible to me that a machine learning model trained to reproduce human text can in principle generate something that is significantly above human text production ability.

These are not the only currently-in-use AI models. However, recent news has even this particular category of AI solving multiple previously unsolved Erdős problems.

(Even if they also do stupid things on a frequent basis).

> Well, not really. For large-scale AI models it's almost exclusively some form of gradient-based non-linear optimization. Genetic algorithms (which is just hill-climbing optimization with extra steps) and genetic programming (which is really cool and not well-understood) do not perform all that well in practice and I'm not aware of any notable applications.

I said "standard technique" rather than "best" for a reason ;)

This is proof-of-possibility: natural selection did it, we know how to mimic that, but we don't know enough to be sure we're doing it with a reward function that will actually give us minds like ours on an interesting timescale with a probability high enough to care about.

[godelski]

  > solving multiple previously unsolved Erdős problems.

So what?

Machines have solved tons of unsolved problems in mathematics. That's not a proof of intelligence.

If you brute force a solution, we congratulate you on your effort.

If you stumble into a solution, we congratulate you for being lucky (if we can distinguish)

If you find a unique solution that no one else imagined, we congratulate you on your intelligence.

These are categorically different things and the difference matters. It's the whole distinction. Though in the real world success usually requires all three (and more), complicating evaluation.

When we're talking about intelligence you can't distill it to "getting the answer". If you do then I'll direct you at an abacus, a calculator, a watch, or google search if you want to look at super intelligence.

  > natural selection did it

Did what? Gradient descent? If that's the argument, you need to read more

[ben_w]

> If you find a unique solution that no one else imagined, we congratulate you on your intelligence.

If that's your definition, bad luck: that is exactly what the AI did.

There are other definitions where AI fail, my example of which would be "how many examples did it take to learn the basics?", ML is as thick as plankton by this definition.

> When we're talking about intelligence you can't distill it to "getting the answer". If you do then I'll direct you at an abacus, a calculator, a watch, or google search if you want to look at super intelligence.

At which point I direct you to the words "combinatorial explosion", how the busy beaver function grows faster than any computable sequence, and how many symbols exist in mathematics.

Valid number of Chess games? Estimated to be between 10^120 and 10^123

Valid number of Go games? Estimated to be between 10^10^108 and 10^10^171

Busy beaver number corresponding to something connected to the Collatz conjecture? BB(15). Nobody knows the exact value of any BB number bigger than BB(5), though we do know the lower bound for BB(7) is > 2↑¹¹↑¹¹3; needing to use up arrow notation is always a sign things are getting wild.

So far as I know, nobody's even bothered to work out what BB number would correspond to the recently solved Erdős problems. But we can look at the size of the lean proofs used for them and say "lol no" to the idea of brute-forcing it inside this universe before heat death gives every particle and photon a wavelength larger than the cosmological horizon.

> Did what? Gradient descent? If that's the argument, you need to read more

If you were to go up the conversation tree two more steps, you would already have the answer:

  Me: We do know* evolution made our brains, and we do use simulated evolution as a standard technique in machine learning.
  defmacr0, replying to "we do use simulated evolution as a standard technique in machine learning.": Well, not really. [says it's limited with examples] in practice and I'm not aware of any notable applications.
  Me: This is proof-of-possibility: natural selection did it

Natural selection made us. That's the answer to "Did what?"

[red75prime]

> ML is as thick as plankton by this definition

...during autoregressive pretraining ([2]). A model pretrained on texts with 1930 data cutoff can solve a few programming problems when given a few examples. Its success rate is understandably much worse.

[1] https://talkie-lm.com/introducing-talkie

[2] That is when a model starts from a blank state that can be described in a few kilobytes.

[ben_w]

Careful, looks like you're shifting definitions.

Remember, I was saying "by this definition", where "this" isn't about capabilities, but about effort needed to get there, which in the case of your link is "260B tokens of historical pre-1931 English text".

A human who read 260e9 tokens will take something like 1900 years of 24/7 reading to get that far, while a more realistic human (though one who still reads a lot) would take 260e9/((50000/0.75)*365) ~= 10,685 years, and that still only gets you something "interesting" rather than "competent".

Don't get me wrong, even merely HumanEval pass@100 ~= 0.04 (eyeballing that chart) shows the model has clearly learned something and isn't just randomly throwing things at the wall. All I'm saying is this took a huge effort to get even that far (and, implicitly, that this is a reasonable argument to use if you want to say they're "not intelligent").

[red75prime]

Autoregressive pretraining (text/images/video prediction) produces a foundational model. You can look at it as a highly compressed conditional probability distribution of the human brain output. The information-theoretically optimal compression of the data is a program that reproduces functionality of the process that generated the data.

So, it stands to reason (and observations) that such a model captures not only surface statistics of the data, but a part of functionality of the system that generated the data (the human brain for text, physics for video).