[djokkataja]

> Nature gives us a world in which "he who does not work, does not eat" is a rule of law.

Which is why no child has ever survived until adulthood.

And TBH I find this an appropriate metaphor because the promise of AGI is that every adult human will be infantile by comparison--but there are also potential technologies that could allow humans to ... "upgrade" themselves and become a mature ... whatever. Sadly brain-computer-interfaces seem like a technology that works best when enabled by AI (to help with interpreting brain signals), so it seems quite unlikely that any biological humans are going to keep up with AI over the medium term.

[Nevermark]

> but there are also potential technologies that could allow humans to ... "upgrade" themselves

I hope so.

But transforming ourselves into GAI to keep up (while maintaining continuity of memories, etc.) is going to be a much more expensive proposition than simply making more GAI hardware from scratch.

So economically, where are humans going to earn that additional value, needed to account for the extra cost? When the premise of upgrading ourselves is our native biology isn't keeping up and so not actually needed?

[djokkataja]

Given that there's no well-defined "unit of intelligence", I see no reason to believe that all AGI (or GAI, to use your acronym) entities will be identical. I also don't think it's plausible to assume that any GAI entity will be literally omniscient; uncertainty remains a reasonable, deep, persistent factor for all entities that don't severely violate physics as we presently understand it.

The additional value that humans present would thus be a diversity factor. Much as it might be economically beneficial to cut down the entire Amazon, there's also significant economic benefits in keeping it intact and finding out what kinds of crazy biological stuff comes out of there. And over the long run, we get way more economic benefit from finding crazy things in there than in chopping it all down. There are all kinds of ways to explore uncertain dimensions, but looking at whatever the universe has already managed remains a consistent source of value.

> But transforming ourselves into GAI to keep up (while maintaining continuity of memories, etc.) is going to be a much more expensive proposition than simply making more GAI hardware from scratch.

It's basically a fixed cost because of how slowly the human population grows (if it even continues to grow).

Synthetic GAI entities will be able to use vastly more resources than humans presently do, because they'll be able to create more of themselves as quickly as resources become available, and they won't be stuck operating at a fixed clock speed. It's also probably going to be much easier for them to make use of off-world resources. So the total population of GAI entities could plausibly explode compared with the total human population.

It comes down to timing: if there's technology available for humans to become less ... biological, and it seems very desirable to lots of people, but it's really expensive and there aren't that many GAI entities yet, that might be less pleasant. But if that kind of tech depends sufficiently on AI developments that there's some significant lag time between a GAI population explosion and that kind of human-upgrading tech showing up, the cost of making it available to whatever humans want it might be a drop in the bucket from a total economic perspective.

To be clear, I don't see biological humans "keeping up" over the long term. If we take AGI and BCIs and various Ship of Theseus questions seriously, there could be some significant blurring of lines between "human" and "AI." And if we combine that with GAIs originating from humans, the concept of "keeping up" seems to become less meaningful. Who's keeping up with who?

Lastly, I'm suspicious that GAI entities won't be purely economically motivated, because I don't see any reason that they'll be "purely" anything at all. There is no magical "essence d'intelligence"; instead there's staggering layers of complexity. And every plausible AI safety approach I've seen so far involves training AIs to be inclined towards beneficial acts and away from harmful acts because there's no way to program conceptually pure "motivations" into them.

[Nevermark]

> I see no reason to believe that all AGI (or GAI, to use your acronym) entities will be identical

I absolutely agree. There will be a Cambrian explosion of superintelligent forms.

> The additional value that humans present would thus be a diversity factor.

We already have a test for this. Parrots, octopus, whales and many other creatures are also diverse. But our problems are beyond their experience. Do we value their diversity as thinkers in our economy?

We don't.

(I am speaking in economic terms. We can deeply intensely highly "value" many things, but if we don't actually invest economic value in protecting or nuturing those things, our appreciation has little or no impact.)

--

> It comes down to timing: if there's technology available for humans to become less ... biological

That makes sence. The order things become viable makes all the difference.

What I am seeing is that AI as software/minds is moving faster and faster. AI "bodies", i.e. robotic forms, are moving much slower but gaining steam. And our ability to augment biological forms is moving far far slower.

All three areas are accelerating, but the disparity between them isn't shrinking - for now. But who knows? The next couple decades will be interesting!

Maybe going fully artificial ("self-designed" might be a more meaningful description) will become trivial sooner than we think. :)

--

> Lastly, I'm suspicious that GAI entities won't be purely economically motivated, because I don't see any reason that they'll be "purely" anything at all.

I think they will be deeply motivated economically to care about ethics. Just as our genes and culture have (slowly) responded to the tremendous value of creating win-win situations.

Also, our general curiosity about about many things that don't have immediate economic value, actually does drive unpredicted personal and society level economic advances over the long run. So I expect they will maintain interest in general and undirected learning.

Similarly, our appreciation for seemingly non-economic shows of creativity (art, music, competitive games, mathematics for its own sake, etc.) can be thought of as the active form of curiosity. The development of novel and surprising artifacts and challenging competitions, may have value to AI's as well.

Everything that is important to us became that way for some economic reason: specifically, survival energy economics. So many of our seemingly non-economic activities may continue to have analogous roles for future intelligences.

[djokkataja]

> We already have a test for this. Parrots, octopus, whales and many other creatures are also diverse. But our problems are beyond their experience. Do we value their diversity as thinkers in our economy?

> We don't.

I don't mean humans as they presently are, but humans as somewhat-novel GAI entities after upgrading (and humans as an ongoing source: likely there will be some humans that aren't interested in changing themselves in such ways, but who might change their mind over time and/or have children who see things differently).

> (I am speaking in economic terms. We can deeply intensely highly "value" many things, but if we don't actually invest economic value in protecting or nuturing those things, our appreciation has little or no impact.)

I don't think this is true. For example, suppose you deeply value a bunch of things that you don't think have much economic significance, but economic significance is the most important thing to your perspective / philosophy, so you don't take care of the things that you deeply value. This does have a significant impact: you become sad, you become depressed, you become less economically productive. So then you say, oh, I'll just take care of those other things I care about, and then I can be economically productive! So you make a few changes but then find yourself feeling down again not long afterwards because economic value is the cornerstone of how you think about things. And then you realize that from an economic perspective, it's more beneficial economically to stick with a perspective that embraces your values as you discern them (and how those continue developing over time) rather than sticking with a perspective that puts economic value first. And then you are taking your first steps to freedom from the curse of your Bachelor's in Economics. ;)

Ethical GAIs also seem likely to me ... hopefully.

[Nevermark]

> suppose you deeply value a bunch of things that you don't think have much economic significance, but [...etc...]

We are a mess. We often think some things are worth more than money, but are not motivated to spend money on them when needed.

Everything form climate change, to saving specific species.

I think GAI's (green field, or human uplifts) will be much more coherent. They won't be settling for a beautiful wonderful Goldberg brain, designed by a series of a million fortunate accidents, like we have to.