[parrellel]

True AI as in a true artificial thinking being? Or true AI as effectively a god?

All comments like this assume the AI is absolutely goal oriented toward self-improvement and self-propagation and has no limitations or competition. "No biology or society" - what about the physical hardware and infrastructure needed to maintain it? "Self upkeep" - so its directly connected to an entire automated vertically integrated supply chain and factory complex? "Unlimited ability to record/analyze" - again, where is it getting this infinite storage and processing capacity? "ability to simulate variations of..." - if it's truly intelligent in a way humans would consider intelligent, even massively moreso, why would it spend all its time random walking the solution space of everything?

If we are talking about AI as god, that can work, I suppose, but as someone who really likes transhumanism and the idea of AI in general, this idea of the sloughing away of all limitations and an AI actor totally dedicated with either the infinite and immediate improvement or replacement of itself makes no sense to me.

Am I missing something here, or is there just to much engineer whispering in the corners of my brain.

[jtolmar]

> an AI actor totally dedicated with either the infinite and immediate improvement or replacement of itself makes no sense to me.

If I decide that a problem is so hard that the best solution is to invent a superhuman AI to solve it, then this is an approach that human-level intelligence can come up with, so a superhuman intelligence can too.

Self-improvement and self-replacement are probably not an AI's actual goal, they're just things that are useful to most potential goals that an AI can have. (And they're easier for the potential AI because the prerequisite research has already been done at that point.)

(If you knew I was trying to either cure cancer or colonize mars, you could predict that I'll start raising money, even though those goals don't have much in common.)

[TuringTest]

> If I decide that a problem is so hard that the best solution is to invent a superhuman AI to solve it, then this is an approach that human-level intelligence can come up with, so a superhuman intelligence can too.

It can try, but it doesn't mean that it would success at it.

[parrellel]

...and the meat-space limitations? Which were the bulk of my point?

[adventured]

> again, where is it getting this infinite storage and processing capacity

I agree with most all of your points. On that one, about where its resources are to come from (and thinking just in a fantastical manner on the topic): it's going to trade with humans, who will provide the things it needs to keep growing. For example, give people a 'free' search engine, that they find highly useful, which results in the humans funding the AI through ads/clicks, allowing for massive expansion of the AI infrastructure. You could apply the general concept to most any service an AI could provide. I'll give you this, you give me that, resulting in a dependency, with both sides occasionally trying to seize on an opening to acquire the upperhand in the arrangement.

Simply put, humans will trade with AI eventually. It's inevitable, and it's a critical aspect of how AI will self-expand. To ensure their own survival, they'll want to become so useful to humans that the humans don't kill (or try to kill) the AI, despite occasionally (or frequently) being afraid of its power / potential.

Want to build out the colonies in the US? You're going to trade with the British Empire and France, while trying not to piss off either enough that they destroy you (or try to). You want to be useful to them, enrich them, limit the extent to which you seem to pose a threat. You maintain an even keel until the point where you are no longer subservient. China has been a master at that the last 40 some years, they still behave that way today when it suits their long-term aims (playing down their strength at times, or playing it up at other times). As a concept, I think this represents a power dynamic that is universal between most living (aware, sentient, whatever) things, and the human-AI relationship will also play out that way.