[roenxi]

It also has an issue not present in Pascal's, which is you are multiplying by infinity in a context where it doesn't actually make sense.

The payoff from AGI may be incalculable, but it isn't infinite both in itself and in Altman's ability to enjoy the rewards it promises. Once the value becomes finite, a whole heap of risk-reward logic kicks in that the Wager wants to sweep under the rug.

As a concrete example, following this Altman's wager would result in Altman giving all his wealth to the first beggar on the street who mumbles that he might be able to run an AGI project - the possibility that the beggar can achieve that is, technically speaking, nonzero. Multiply that by infinity and you have a great expected return (infinite, in fact). However, practically speaking, the risk will overwhelm the large-but-finite payoff.

Infinity is bigger than people think :P

[mcguire]

According to the true believers, the payoff for AGI is infinite because the superintelligence will be capable of literally anything (or at least simulating it well enough that it doesn't matter). To them, it is Pascal's wager.

Religious beliefs can be weird.

[lisper]

Two other possibilities that Altman's wager as presented does not take into account:

1. The government could change the rules (i.e. redistribute wealth) in response to the new economic reality brought about by AGI

2. The AGI may decide to retain the economic value it captures for itself