[guscost]

You're basing all of this on one heck of an axiom:

> The danger with AI is that it grows in power exponentially

This is like saying "the opportunity with mechanical transportation is that it gets faster exponentially" before even inventing the wheel.

[TheOtherHobbes]

Yes, exactly. I think faith in an apocalyptic singularity is a hangover from religion, and not something that can be justified scientifically.

We're actually incredibly bad at making robust, reliable software. So there's no realistic basis for assuming a self-improving machine is even possible. Never mind a conscious self-improving machine. Even less a conscious self-improving machine that develops god-like capabilities at an exponential rate.

Game changer tech is always possible. But AI-on-silicon is going to be a dead end without some new non-Turing computing substrate.

The real problems are political and social, and we already have those. Automation - rather than true autonomous AGI - may well make them worse. But that's a different problem, and not obviously related to quasi-sentient paperclip machines rampaging through our cities.

[whack]

You've missed the point. Mechanical transportation are dependent on other entities (ie, humans) for progress. There's no positive feedback loop. AGI is fundamentally different because an AGI can design an even better AGI, thus producing positive feedback loops, and exponential growths.

[guscost]

It sounds like you've missed my point as well. The analogy is only supposed to illustrate that before actually inventing transportation technology (which for a long time did get faster exponentially), humans had no real basis to understand the tradeoffs inherent in rolling vehicles, floating vehicles, flying vehicles, impulse/rocket vehicles, etc. Neither did they share our current understanding of a theoretical maximum speed that anything can ever go according to physics.

> AGI is fundamentally different because an AGI can design an even better AGI

Thanks for pointing this out, but while I think I understand the distinction ("AGI is technology that works like humans, and since humans can design better technology, an AGI can design better versions of itself") that statement also relies on several axioms:

1. Humans can design a general intelligence.

2. A general intelligence can exist in a stable state with a fundamentally "better" design than ours (i.e. one that can be exponentially more powerful, not just a bit better at poetry).

3. A general intelligence can improve itself and/or design better versions of itself without hitting diminishing returns, or it can design a fundamentally better version of itself from scratch if that happens.

It's fine if you believe all of those things, and I guess lots of people do, but I wouldn't just sweep those axioms under a blanket statement about AGI designing better AGI unless you know that everyone agrees with them.

[red75prime]

To escape a bear you don't need to run faster than the bear, you just need to run faster than a friend.

AI doesn't need to be exponentially self-improving to pose a threat to humanity. It needs to improve faster than humanity.

[red75prime]

Every day you are looking at the progress achieved by the system consisting of generally intelligent agents, who are unable to upgrade, who have different goals, who are limited in their communication speed, who can't copy themselves.