[torginus]

You want to go to the store to get ice cream. Ice cream is delicious and the value of eating ice cream is a small positive, let's say x. There's a one in ten million chance you'll get hit by a car on the way and die, and your life is infinitely precious, therefore the expected value of going is x times 1 = x, and the one of not going is 1/10m times negative infinity which is negative infinity. You are a rational person, so you don't go. In fact you don't do much of anything. Your value model of every activity has collapsed to a single value.

That's the problem with 'singularity' arguments. The people making them ignore the fact that the mathematical definition of the word means 'the model of outcomes collapses to a single value' therefore the model stops being useful, yet they somehow claim to be able to make predictions beyond the singularity. It's like those shitty Facebook math posts where they divide both sides of the equation by 0 (the fact hidden by some sleight of hand), to 'prove' that 2=1.

The formulation of the singularity involves putting outrageous values into the parameters of the model of reality, and denominator ignorance, and then claiming 'rationally' determining that the consequences are too severe to ignore.

[Balgair]

Aside:

The singularity framing is really tough here, right? It comes from black hole physics. Essentially, at the event horizon, the way we know how to do physics stops working, and we rightly conclude that we can't currently say anything about the other side of the event horizon. It is not saying that nothing is occurring there. Matter, time, space, energy, whatever, that still is there (maaaaybe?) and is still undergoing something. It's just that we don't know what that is.

The same is true with using these tech singularity arguments. Like, in the age of superintelligence (if that happens), there will still be thing happening, the dawn will still come every day and the dusk will still too. It's just that we say our current ideas about that new day aren't that applicable to that new age (God, this sounds like a hippie).

However, unlike with black hole physics where we aren't even sure time can exist like we know, we are likely all going to be there in that new superintelligence age. We're still going to be making coffee and remembering bad cartoons from our youth. Like, the analogy to black hole physics breaks down here and maybe does a disservice to us. It's not a stark boundary at the Schwartzchild radius, it is a continuous thing, a messy thing, a volatile thing, and very importantly for the HN userbase, a thing that we control and have the choice to participate in.

We are not passively falling into the AGI world like the gnawing grinding gravity of a black hole.

[muvlon]

I don't know, are we actually going to be making coffee in the case of the AGI singularity?

If you listen to the hardcore doomers, the misaligned superintelligence will curl a finger on its monkey paw and turn the planet into paperclips or something. If you listen to the most depraved boosters, AGI will remove the need for 99.999% of human workers and so we all get turned into biofuel to churn out more tokens.

Yes those are really extremely scenarios but that's how I think of the singularity. It's so alien that we cannot rule out anything.

[_alternator_]

The event horizon = singularity metaphor is a little off. There is no breakdown in the laws of physics at the event horizon. It's just that there is no light or matter that escapes from the event horizon. But the laws of physics don't break down until you reach the center of the black hole (which will happen in finite time after you cross the event horizon).

So there are a couple interesting and meaningful changes at the event horizon, but it's not a mathematical singularity.

[uselessTA]

There's an objection here when you get to tiny numbers, but surely you wouldn't get ice cream if there was a 10+% chance to get hit by a car?

I think he's saying that >1% or even >10% chunk of your probability mass should be on superintelligence, otherwise you're implicily >99% confident of stalled progress, which seems overconfident. We're not talking about not some infinitesimal fraction here.

[metalcrow]

And yet sometimes the consequences ARE too severe to ignore. Nuclear war is a serious concept and it's carefully investigated and attempted to be controlled by a lot of powerful people. Why is this situation different? Because it's unlikely? So is nuclear war.