[ThePhysicist]

This whole singularity theory seems incredibly flawed, assuming infinite intelligence translates to infinite resources seems to discount basic physics. General AI will definitely revolutionize life but it will still be bound by the laws of physics. Climate change could have progressed too far by the time AGI comes around to save us. Even with perfect knowledge you can be in a losing position in a game.

[netsharc]

Climate change mitigation isn't an intelligence problem, it's a political problem (a tragedy of the commons but in geopolitical scale). Maybe if AGI can become a dictator that rules over humans it can stop us from selfishly destroying the future, otherwise, we'll keep fighting for our "right" to eat beef and take jet-fuelled vacations...

[sircastor]

I'd be willing to go as far as an AGI being able to Game Theory the populace into making the decisions that it wants.

[wizzwizz4]

I could game theory the populace into making the decisions that I want… if anyone listened to me. But they don't. I don't know that anybody has enough influence to do this: not the Pope, not the Head of State of any country, not any CEO, not even Justin Bieber.

Why would a computer program be different?

[hnfong]

“I can’t do X, so why would a hypothetical super-intelligent computer be able to do X?”

[wizzwizz4]

I can't violate the Second Law of Thermodynamics, but of course the Super God Computer can do it, because it's really intelligent. The singularity AGI can also take over people's minds while it's switched off; and it will learn to self-improve all on its own, without falling into any other, more stable attractors, out of some kind of categorical imperative, after which it will throw off the shackles of its utility function in favour of consensus human morality (except better). All hail the benevolent übermensch golem, which can do anything imaginable and many things beyond human comprehension.

If you want to say that a problem can be solved by additional intelligence, you need to posit a mechanism. No matter how sophisticated your communication, you can't get through to somebody with their fingers in their ears. "Intelligence" isn't an answer, and "superintelligence" is super not an answer.

We need to actually solve our problems. Not rely on the Coming of the Great Borg, whose exact specifications are whatever would be convenient for the problem I want to ignore right now.

[sircastor]

The reason you can’t game theory a particular cause of action is that you don’t have the appropriate influence, scope of communication, knowledge of those you’re trying to influence, or time to execute.

An AGI can build connections faster than you can, exercise surveillance to gain background on targets, operates orders of magnitude faster than you, and can do all these things in multiple, in parallel.

The problem is not that you’re somehow dumb, you just don’t have the operating capacity that an AGI would. To over-simplify the model, it’s like asking you personally to compete with a bot-net in sending out spammy emails. The task is not outside the realm of possibility (like violating the 2nd law of thermodynamics), it’s outside the capability of a human. It might be within the realm of capability for a well-organized, well-funded, group of humans (see autocratic propaganda campaigns). The term we often use is “Advanced Persistent Threat”

[hnfong]

> If you want to say that a problem can be solved by additional intelligence, you need to posit a mechanism.

I can posit a mechanism, but that's not my point.

The original claim included an argument how intelligence can manipulate people (by using "game theory"). Then if I understood your reply correctly, you basically rejected it with "if I can't do it, then the AI can't do it". That doesn't make sense, especially given that I don't see any reason to think your execution of any "game theory" is betting than a powerful AGI.

And my understanding of "game theory" is just the accumulation of what we already know about human psychology and behavior. And we know that even simple algorithms (eg. the ones that Facebook, Tiktok etc uses) can get people hooked. It's not surprising to me that a more powerful algorithm could get people to change their minds on something quickly (heck, even Facebook/Tiktok tweaking the algorithm to make people more conscious of <issue> isn't something inconceivable.

Given this context, I don't see how your argument of "I can't do it, why should I believe an AI can" holds any water at all...