[T-A]

The scenario usually bandied about is AGI self-improving at an accelerating rate: once you cross the threshold to self-improvement, you quickly get superintelligence with God-like powers beyond human comprehension (a.k.a. the Singularity) as AGI v1 creates a faster AGI v2 which creates a faster AGI v3 etc.

Any AI researchers still plodding along at mere human speed are then doomed: they won't be able to catch up even if they manage to reproduce the original breakthrough, since the head start enjoyed by AGI #1 guarantees that its latest iteration is always further along the exponential self-improvement curve and therefore superior to any would-be competitor. Being rational(ists), they give up and welcome their new AI overlord.

And if not, the AI god will surely make them see the error of their ways.

[true_religion]

What if AI self improvement is not exponential?

We assume a self improving AI will lead to some runaway intelligence improvement but if it grows at 1% per year or even per month that’s something we can adapt to.

[devbent]

Assume the AGI has access to a credit card and it goes ahead and reserves itself every GPU cycle in existence so it's 1 month is turned into a day, and now we're back to being fucked.

Maybe an ongoing GPU shortage is the only thing that'll save us!

[sicariusnoctis]

How would an AGI gain access to an unlimited credit card that immediately gives it remote access to all GPUs in the world?

[JetSpiegel]

It could hack into NVIDIA and AMD, compromise their firmware build machines silently, then publish a GPU vulnerability that required firmware updates.

After a couple months, turn on the backdoor.

[veidr]

E.g. by convincing 35% of this website's users to "subscribe" to its "service"?

¯\_(ಠ_ಠ)_/¯

[anonymouskimmer]

It seems to me that non-General AI would typically outcompete AGI, all else held equal. In such a scenario even a first-past-the-post AGI would have trouble becoming an overload if non-Generalized AIs were marshaled against it.

[Aerbil313]

This makes no sense at all.

[anonymouskimmer]

https://news.ycombinator.com/item?id=38335012

[veidr]

uhm, wat?

[anonymouskimmer]

This is just the thesis that paperclip optimizers win over general intelligence, because they optimize.