[throwaway0123_5]

I'm not completely convinced of this, even in the presence of AGI that is peak-human intelligence in all ways (lets say on-par with the top 1% researchers from top AGI labs, with agency and online learning are fully solved). One reason for this is what the sibling comment argues:

> Exponentially smarter AI meets exponentially more difficult wins.

Another is that it doesn't seem like intelligence is the main/only bottleneck to producing better AIs right now. OpenAI seems to think building a $100-500B data center is necessary to stay ahead*, and it seems like most progress thus far has been from scaling compute (not to trivialize architectures and systems optimizations that make that possible). But if GPT-N decides that GPT-N+1 needs another OOM increase in compute, it seems like progress will mostly be limited by how fast increasingly enormous data centers and power plants can be built.

That said, if smart-human-level AGI is reached, I don't think it needs to be exponentially improving to change almost everything. I think AGI is possibly (probably?) in the near-future, also believing that it won't improve exponentially doesn't ease my anxiety about potential bad outcomes.

*Though admittedly DeepSeek _may_ have proven this wrong. Some people seem to think their stated training budget is misleading and/or that they trained on OpenAI outputs (though I'm not sure how this would work for the o models given that they don't provide their thinking trace). I'd be nervous if it was my money going towards Stargate right now.

[DennisP]

Well we do have an existence proof that human-level intelligence can be trained and run on a few thousand calories per day. We just haven't figured out how to build something that efficient yet.

[snackbroken]

The inference and on-line fine tuning stage can run on a few thousand calories a day. The training stage has taken roughly 100 TW * 1bn years ≈ 10²⁸ calories.

[DennisP]

Hmm I'm not convinced that human brains have all that much preprogrammed at birth. Babies don't even start out with object permanence. All of human DNA is only six billion bits, which wouldn't be much even if it encoded neural weights instead of protein structures.

[snackbroken]

Human babies are born significantly premature as a compromise between our upright gait and large head-to-body ratio. A whole lot of neurological development that happens in the first couple of years is innate in humans just like in other mammals, the other mammals just develop them before being born. E.g. a foal can walk within hours of being born.

Babies are born with a fully functioning image recognition stack complete with a segmentation model, facial recognition, gaze estimator, motion tracker and more. Likewise, most of the language model is pre-trained and language acquisition is in large part a pruning process to coalesce unused phonemes, specialize general syntax rules etc. Compare with other animals that lack such a pre-trained model - no matter how much you fine-tune a dog, it's not going to recite Shakespeare. Several other subsystems come online in the first few years with or without training; one example that humans share with other great apes is universal gesture production and recognition models. You can stretch out your arm towards just about any human or chimpanzee on the planet and motion your hand towards your chest and they will understand that you want them to come over. Babies also ship with a highly sophisticated stereophonic audio source segmentation model that can easily isolate speaking voices from background noise. Even when you limit yourself to just I/O related functions, the list goes on from reflexively blinking in response to rapidly approaching objects to complicated balance sensor fusion.

[DennisP]

If you're claiming that humans are born with more data than the six gigabits of data encoded in DNA, then how do you think the extra data is passed to the next generation?

[snackbroken]

I'm not claiming that humans are somehow born with way more than a few billion parameters, no. I'm agreeing that we have an existence proof for the possibility of an efficient model encoding that only requires a few thousand calories to run inference. What we don't have is an existence proof that finding such an encoding can be done with similar efficiency because the one example we have took billions of years of the Earth being irradiated with terawatts of power.

Can we do better than evolution? Probably; evolution is a fairly brute force search approach and we are pretty clever monkeys. After all, we have made multiple orders of magnitude improvements in the state of the art of computations per watt in just a few decades. Can we do MUCH better than evolution at finding efficient intelligences? Maybe, maybe not.

[pdimitar]

I agree with your take and would slightly refine it to remark that having in mind how protein unfolding / producing works in our bodies, I'd say our genome is heavily compressed and we can witness decompression with an electronic microscope (how RNA serves like a command sequence determining the resulting protein).

[apitman]

The human genome has 6 billion bases, not 6 billion bits. Each base can take one of 4 values, so significantly more data than binary. But maybe not enough of a difference to affect your point.

[DennisP]

Looks like actually three billion base pairs in human DNA: https://www.genome.gov/genetics-glossary/Base-Pair#:~:text=O...

So six billion bits since two bits can represent four values. Base pairs and bases are effectively the same because (from the link) "the identity of one of the bases in the pair determines the other member of the pair."

[apitman]

It's 6 billion because you have 2 copies of each chromosome. So 12 billion bits right? But I do think your original point stands. I'm mostly being pedantic.