[notahacker]

Has the same issue as Pascal's. Competing AGI projects (gods) exist and their believers might be the ones reaping the infinite rewards, not to mention the distinct possibility the AGI (god) doesn't actually see rewarding believers as its highest priority, and might choose to share its infinite rewards with people who aren't part of the inner circle, or even punish people who joined inner circles with the express intent of elevating themselves above ordinary people :)

Actually makes a bit more sense from the traditional view that AGI projects might not reach their goal but the well run ones are likely to have very commercially valuable byproducts anyway. If we were getting Star Trek economics out of it, who'd be interested in entirely obsolete concepts like "economic value" and "100x returns" anyway?

[roenxi]

It also has an issue not present in Pascal's, which is you are multiplying by infinity in a context where it doesn't actually make sense.

The payoff from AGI may be incalculable, but it isn't infinite both in itself and in Altman's ability to enjoy the rewards it promises. Once the value becomes finite, a whole heap of risk-reward logic kicks in that the Wager wants to sweep under the rug.

As a concrete example, following this Altman's wager would result in Altman giving all his wealth to the first beggar on the street who mumbles that he might be able to run an AGI project - the possibility that the beggar can achieve that is, technically speaking, nonzero. Multiply that by infinity and you have a great expected return (infinite, in fact). However, practically speaking, the risk will overwhelm the large-but-finite payoff.

Infinity is bigger than people think :P

[mcguire]

According to the true believers, the payoff for AGI is infinite because the superintelligence will be capable of literally anything (or at least simulating it well enough that it doesn't matter). To them, it is Pascal's wager.

Religious beliefs can be weird.

[lisper]

Two other possibilities that Altman's wager as presented does not take into account:

1. The government could change the rules (i.e. redistribute wealth) in response to the new economic reality brought about by AGI

2. The AGI may decide to retain the economic value it captures for itself

[arugulum]

>Has the same issue as Pascal's.

I get the feeling that some people (maybe not this commenter) are missing the point I'm making.

Yes, Pascal's wager is flawed - which is why I'm modeling the Altman's wager after it.

I thought I had made the wording and respective payoff statements sufficiently tongue-in-cheek, but I guess this is Poe's law biting back.

[notahacker]

tbf I wasn't 100% sure whether you were tongue-in-cheek or had just spent way too much time taking variants of this argument seriously on LessWrong...

(also, as much as I'm also tongue in cheek when talking about AGIs punishing [infinite numbers of simulations of] Sam Altman for backing the wrong AGI project, I'm dead serious about them potentially being able to turn a decent profit selling advanced data processing to Silicon Valley startups. And let's face it, Sam's a much better CEO to be able to help them achieve that than AI Rapture anyway)

[kristianov]

Agreed. It seems extremely naive that they believe they can control the power of AGI, plus it is borderline ridiculous that their goal for AGI is to capture economic value.

[throwawaymath]

I'm skeptical of OpenAI, but to be charitable I don't think economic value is the primary goal for their AGI. Their position seems to be that it needs to be one of several necessary goals in order to repay investors for helping them scale up to achieve it in the first place.