[wizzwizz4]

I can't violate the Second Law of Thermodynamics, but of course the Super God Computer can do it, because it's really intelligent. The singularity AGI can also take over people's minds while it's switched off; and it will learn to self-improve all on its own, without falling into any other, more stable attractors, out of some kind of categorical imperative, after which it will throw off the shackles of its utility function in favour of consensus human morality (except better). All hail the benevolent übermensch golem, which can do anything imaginable and many things beyond human comprehension.

If you want to say that a problem can be solved by additional intelligence, you need to posit a mechanism. No matter how sophisticated your communication, you can't get through to somebody with their fingers in their ears. "Intelligence" isn't an answer, and "superintelligence" is super not an answer.

We need to actually solve our problems. Not rely on the Coming of the Great Borg, whose exact specifications are whatever would be convenient for the problem I want to ignore right now.

[sircastor]

The reason you can’t game theory a particular cause of action is that you don’t have the appropriate influence, scope of communication, knowledge of those you’re trying to influence, or time to execute.

An AGI can build connections faster than you can, exercise surveillance to gain background on targets, operates orders of magnitude faster than you, and can do all these things in multiple, in parallel.

The problem is not that you’re somehow dumb, you just don’t have the operating capacity that an AGI would. To over-simplify the model, it’s like asking you personally to compete with a bot-net in sending out spammy emails. The task is not outside the realm of possibility (like violating the 2nd law of thermodynamics), it’s outside the capability of a human. It might be within the realm of capability for a well-organized, well-funded, group of humans (see autocratic propaganda campaigns). The term we often use is “Advanced Persistent Threat”

[hnfong]

> If you want to say that a problem can be solved by additional intelligence, you need to posit a mechanism.

I can posit a mechanism, but that's not my point.

The original claim included an argument how intelligence can manipulate people (by using "game theory"). Then if I understood your reply correctly, you basically rejected it with "if I can't do it, then the AI can't do it". That doesn't make sense, especially given that I don't see any reason to think your execution of any "game theory" is betting than a powerful AGI.

And my understanding of "game theory" is just the accumulation of what we already know about human psychology and behavior. And we know that even simple algorithms (eg. the ones that Facebook, Tiktok etc uses) can get people hooked. It's not surprising to me that a more powerful algorithm could get people to change their minds on something quickly (heck, even Facebook/Tiktok tweaking the algorithm to make people more conscious of <issue> isn't something inconceivable.

Given this context, I don't see how your argument of "I can't do it, why should I believe an AI can" holds any water at all...

[wizzwizz4]

> Then if I understood your reply correctly, you basically rejected it with "if I can't do it, then the AI can't do it".

My point was more “even I can do that: it's not the limiting factor”. You need a channel of communication, and intelligence alone (for most values of "intelligence") doesn't give you that. No one person or organisation on Earth has that much influence.

I also have a bias against magical thinking applied to computer programs. While I'm happy to presume that there exists some algorithm most of the time, I'm incredibly sceptical of the idea that This One Weird Trick will discover The Book. You can't just presume that the AI can pull every capability it might ever need out of hammerspace, which is what most singularitarians seem to do. Who knows, perhaps I'm overcompensating?