[astrange]

> Is there a law of thermodynamics which prevents AI from writing code which would train a better AI?

You need to apply Wittgenstein here.

This appears to be true because you haven't defined "better". If you define it, it'll become obvious that this is either false or true, but if it is true it'll be obvious in a way that doesn't make it sound interesting anymore.

(For one thing our current "AI" don't come from "writing code", they just come from training bigger models on the same data. For another, making changes to code doesn't make it exponentially better, and instead breaks it if you're not careful.)

> I guess people working there believe in magic.

Yes, OpenAI was literally founded by a computer worshipping religious cult.

> People believe that self-improvement might happen when AI is around human-level.

Humans don't have a "recursive self-improvement" ability.

Also not obvious that an AI that was both "aligned" and "capable of recursive self-improvement" would choose to do it; if you're an AI and you're making a new improved AI, how do you know it's aligned? It sounds unsafe.

[Nevermark]

> Humans don't have a "recursive self-improvement" ability

They do.

Humans can learn from new information, but also by iteratively distilling existing information or continuously optimizing performance on an existing task.

Mathematics is a pure instance of this, in the sense that all the patterns for conjectures and proven theorems are available to any entity to explore, no connection to the world needed.

But any information being analyzed for underlying patterns, or task being optimized for better performance, creates a recursive learning driver.

Finally, any time two or more humans compete at anything, they drive each other to learn and perform better. Models can do that too.

[alanbernstein]

> they just come from training bigger models on the same data

Are you arguing that all AI models are using the same network structure?

This is only true in the most narrow sense, looking at models that are strictly improvements over previous generation models. It ignores the entire field of research that works by developing new models with new structures, or combining ideas from multiple previous works.

[astrange]

I sure am ignoring that, because the bitter lesson of AI is usually applicable and implies that all such research will be replaced by larger generic transformer networks as time goes on.

The exception is when you care about efficiency (in training or inference costs) but at the limit or if you care about "better" then you don't.

[recursivecaveat]

This is kindof an odd statement because the transformer is not the most generic neural net. It's the result of many levels of improvements in architecture over older designs. The bitter lesson is methods that can scale well with compute win (alpha/beta beats heuristics alone, neural network beats alpha/beta), not that the most obvious and generic approach eventually wins. Given the context-length problems with transformers I think it's fair to say they have scaling problems.

[killerstorm]

There's a principle more powerful than the bitter lesson: GIGO.

Training to predict internet dump can only give you so much.

There's a paper called something like "learning from textbooks" where they show that a small model trained on high-quality no-nonsense dataset can beat a much bigger model at a task like Python coding.

[Tenobrus]

it is very clear to me that humans do in fact have a recursive self-improvement ability, and i'm confused why you think otherwise

[astrange]

I think people can read books (self improvement) and have children (recursive), but neither of those are both.

[lucubratory]

Why do you think that the human population is more intelligent, knowledgeable, and achieves greater technological feats as time goes on? It's because of recursive self-improvement, we are raised and educated into being better in a quite general sense, which includes being better at raising and educating; nearly every generation this cycle repeats and has for all of human history, at least since we acquired language. We also build machines that help us to make better machines, and then we use those better machines to make even better machines, another example of recursive self-improvement.

[webmaven]

You're pointing out that groups/institutions/cultures/civilizations are examples of recursively self-improving entities, but the original point was about a recursively self-improving individual intelligent entity.

Well, to the extent that a human-level intelligence is an individual, anyway. We ourselves are probably a mixture-of-experts in some sense.

[lucubratory]

An individual human starts out a mewling baby and can end up a maxillofacial surgeon through at least partial examples of recursive self-improvement. Learn to walk, talk, read, write, structure, argue, essay, study, cite etc all the way through to the end, with what you previously learned allowing you to learn even more. There's a huge amount of outside help, but at least some of it is also self-improvement.

Also, for the purposes of talking about the phenomenon of recursive self-improvement, individual vs society isn't the end of analysis. Part of the reason AI recursive self-improvement is concerning is that people are worried about it happening on much faster than societal timescales, in ways that are not socially tractable like human societies are (e.g. if our society is "improving" in a way we don't like, we or other humans can intervene to prevent, alter, or mitigate it). It's also important to note that when we're talking about "recursive self-improvement" when it comes to AI, the "self" is not a single software artifact like Llama-70B. The "self" is AI in general, and the most common proposed mechanism is that an AI is better than us at designing and building AIs, and the resulting AI it makes us even better at designing and building AIs.

[rralian]

New generations build onto the scientific knowledge of previous generations. It may not be fast but that sounds like recursive improvement to me. It seems reasonable for AI to accelerate this process.

[astrange]

I think saying all of society is doing it is plausible, but not the same thing as a single human or AI doing it.

Though… still don't think it's true. Isn't "society is self improving" what they call Whig history?

[killerstorm]

AI might have multiple instances within a single computing environment, so it's more like a population than a single individual.

I.e. "You can only use the memory which you currently use" would be a weird artificial constraint not relevant in practice.

[spacecadet]

A very small percentage maybe. I think I agree with the notion that most people bias toward thinking they are improving while actually self-sabotaging.

[stale2002]

> If you define it, it'll become obvious that this is either false or true

Ok. So then I guess it isn't "just a belief that magic".

Instead, it is so true and possible that you think it is actually obvious!

I'm glad you got convinced in a singular post that recursive self improvement, in the obvious way, is so true and real that it is obviously true and not magic.

[killerstorm]

> This appears to be true because you haven't defined "better".

Better intelligence can be defined quite easily: something which is better at (1) modeling the world; (2) optimizing (i.e. solving problems).

But if that would be too general we can assume that general reasoning capability would be a good proxy for that. And "better at reasoning" is rather easy to define. Beyond general reasoning better AI might have access to wider range of specialized modeling tools, e.g. chemical, mechanical, biological modeling, etc.

> if it is true it'll be obvious in a way that doesn't make it sound interesting anymore.

Not sure what you mean. AI which is better at reasoning is definitely interesting, but also scary.

> they just come from training bigger models on the same data.

I don't think so. OpenAI refuses to tell us how they made GPT-4. I think a big part of it was preparing better, cleaner data sets. Google tells us that specifically improved Gemini's reasoning using specialized reasoning datasets. More specialized AI like AlphaGeometry use synthetic datasets.

> Yes, OpenAI was literally founded by a computer worshipping religious cult.

Practice is the sole criterion for testing the truth. If their beliefs led them to better practice then they are closer to truth than whatever shit you believe in. Also I see no evidence of OpenAI "worshipping" anything religion-like. Many people working there are just excited about possibilities.

> Humans don't have a "recursive self-improvement" ability.

Human recursive self-improvement is very slow because we cannot modify our brains' at will. Also spawning more humans takes time. And yet humans made huge amount of progress in the last 3000 years or so.

Imagine that instead of making a new adult human in 20 years you could make one in 1 minute with full control over neural structures, connections to external tools via neural links, precisely controlled knowledge & skills, etc.

[always2slow]

>> I guess people working there believe in magic.

>Yes, OpenAI was literally founded by a computer worshipping religious cult.

What cult is this?

[astrange]

HPMOR readers who live in group home polycules in Berkeley who think they need to invent a good computer god to stop the evil computer god.

[killerstorm]

You're confusing OpenAI and MIRI.

OpenAI founders: Ilya Sutskever, Greg Brockman, Trevor Blackwell, Vicki Cheung, Andrej Karpathy, Durk Kingma, Jessica Livingston, John Schulman, Pamela Vagata, Wojciech Zaremba, Sam Altman. All of them come from software tech industry and academic research circles, not evidence of interest in HPMOR or Yud.

[Vecr]

I think they cleaned out some of the EAs around the time of the board situation, but I don't know what the non-EA overlap is with your description.