[naasking]

Firstly, growth rates of non-information tech largely isn't relevant to AI. That said, overall growth in knowledge is at least directly proportional to population growth. Even pessimistically considering humans as dumb, an exponential population growth means an exponential discovery rate just from sheer trial and error. So overall progress is undeniably exponential, even if it's a low exponent.

Secondly, we know definitively that information density has been growing exponentially given Moore's law. The much decried end of Moore's law is for a particular incarnation of information tech, but there's still plenty of room to grow in other directions.

Even with our current tech base, we can continue to scale exponentially in horizontal directions with more parallelism (see the rise of core counts, GPU and distributed computing). We're nowhere near the end of that scaling in that direction, let alone longer term innovations like optical and quantum computing.

So really, what possible reasons do we really have for thinking that exponential growth will not continue well past human intelligence? Note, I didn't say infinitely, just well past our intelligence.

[aetherson]

Overall growth in knowledge isn't at least directly proportional to population growth, if the proportion of knowledge already known and shared grows with population growth.

But even if it did, we don't have another doubling of human population ahead of us, so you better hope we're already there.

As you point out, Moore's Law doesn't have a ton more power available to it either.

Lots of problems don't parallelize well, quantum computing has never demonstrated more power than classical computing, and who knows where optical computing will go, but more to the point, hardware growth doesn't in fact guarantee an intelligence explosion.

What possible reasons do we have for thinking that exponential growth has ever happened in terms of actual progress, rather than things like "transistor density"?

Look, every futurist in the world in 1975 thought that by 2017, we'd all be routinely traveling faster than sound, that we'd have colonies on the moon if not mars, and that probably we'd have AGI or something pretty close to it by 2017. The reasons we don't have supersonic travel and common space travel aren't simplistic things like "it's physically impossible to pack energy this densely" or "you can't go this fast."

[naasking]

> Overall growth in knowledge isn't at least directly proportional to population growth, if the proportion of knowledge already known and shared grows with population growth.

I don't see why.

> But even if it did, we don't have another doubling of human population ahead of us, so you better hope we're already there

Right, but we have plenty of doublings of intelligent non-human agents ahead of us. Until then, we increase our effective intelligence using semi-intelligent machines, like we've been augmenting our physical strength with mechanical devices for millennia.

> As you point out, Moore's Law doesn't have a ton more power available to it either.

I disagree. Frequency scaling won't yield too much more improvement. There are other scaling modes available though, as I described. Moore's law is about information density, not performance.

> Lots of problems don't parallelize well

Often repeated, but frequently overstated. Our knowledge of parallelism is still in its infancy.

> quantum computing has never demonstrated more power than classical computing,

Any other option would require rewriting a lot of physics.

> hardware growth doesn't in fact guarantee an intelligence explosion.

Increased information density beyond that available in the human brain means simulating said brain is feasible. That's as close to a guarantee as you can get.

> Look, every futurist in the world in 1975 thought that by 2017, we'd all be routinely traveling faster than sound, that we'd have colonies on the moon if not mars, and that probably we'd have AGI or something pretty close to it by 2017

Except I'm not giving a timeline, I'm saying it's inevitable. Low exponent exponential growth is still exponential. The intelligence explosion is about a trend, not a fixed milestone.