[whack]

The danger with AI is that it grows in power exponentially, especially since highly advanced AI can start improving on themselves without human intervention. When people think exponential curves, they think rapid progress, but that's only half the story. Any exponential curve starts off with looking like a flat line, before suddenly taking off like a rocket ship.

Without the benefit of hindsight, we can't tell how far away we are from that rocket-ship liftoff. We've had decades of minor progress in the past, but that's normal for any exponential curve. Are we going to have many more decades/centuries to go before we get to the breakout moment? Or is it just 10-20 years away? We have no idea. All we know is that once we get to that point, AI-IQ is going to grow exponentially faster than natural human IQ.

That said, I really don't think that censoring AI research is going to work. Pandora's box has been opened, and if we don't do it, someone else will. All this talk about hard coding Asimov's laws into AIs is idiotic as well. We have no clue how to build AGI right now, and until we do, discussing specific tactics like the above is utterly pointless. They also presuppose human ability to shackle and mold super-intelligent beings, without making any mistakes or overlooking unintended consequences, which is nothing more than a pipe dream.

Realistically, there's only one thing we can do. Embrace bioengineering. Embrace GATTACA style genetic selection. Embrace cybernetic augmentation. Do everything we can to grow our IQ beyond its natural limits. If our minds don't keep up with technological progress, we will inevitably find ourselves left behind.

[guscost]

You're basing all of this on one heck of an axiom:

> The danger with AI is that it grows in power exponentially

This is like saying "the opportunity with mechanical transportation is that it gets faster exponentially" before even inventing the wheel.

[TheOtherHobbes]

Yes, exactly. I think faith in an apocalyptic singularity is a hangover from religion, and not something that can be justified scientifically.

We're actually incredibly bad at making robust, reliable software. So there's no realistic basis for assuming a self-improving machine is even possible. Never mind a conscious self-improving machine. Even less a conscious self-improving machine that develops god-like capabilities at an exponential rate.

Game changer tech is always possible. But AI-on-silicon is going to be a dead end without some new non-Turing computing substrate.

The real problems are political and social, and we already have those. Automation - rather than true autonomous AGI - may well make them worse. But that's a different problem, and not obviously related to quasi-sentient paperclip machines rampaging through our cities.

[whack]

You've missed the point. Mechanical transportation are dependent on other entities (ie, humans) for progress. There's no positive feedback loop. AGI is fundamentally different because an AGI can design an even better AGI, thus producing positive feedback loops, and exponential growths.

[guscost]

It sounds like you've missed my point as well. The analogy is only supposed to illustrate that before actually inventing transportation technology (which for a long time did get faster exponentially), humans had no real basis to understand the tradeoffs inherent in rolling vehicles, floating vehicles, flying vehicles, impulse/rocket vehicles, etc. Neither did they share our current understanding of a theoretical maximum speed that anything can ever go according to physics.

> AGI is fundamentally different because an AGI can design an even better AGI

Thanks for pointing this out, but while I think I understand the distinction ("AGI is technology that works like humans, and since humans can design better technology, an AGI can design better versions of itself") that statement also relies on several axioms:

1. Humans can design a general intelligence.

2. A general intelligence can exist in a stable state with a fundamentally "better" design than ours (i.e. one that can be exponentially more powerful, not just a bit better at poetry).

3. A general intelligence can improve itself and/or design better versions of itself without hitting diminishing returns, or it can design a fundamentally better version of itself from scratch if that happens.

It's fine if you believe all of those things, and I guess lots of people do, but I wouldn't just sweep those axioms under a blanket statement about AGI designing better AGI unless you know that everyone agrees with them.

[red75prime]

To escape a bear you don't need to run faster than the bear, you just need to run faster than a friend.

AI doesn't need to be exponentially self-improving to pose a threat to humanity. It needs to improve faster than humanity.

[red75prime]

Every day you are looking at the progress achieved by the system consisting of generally intelligent agents, who are unable to upgrade, who have different goals, who are limited in their communication speed, who can't copy themselves.