[AnthonyMouse]

> Therefore, nothing can produce greater positive expected value than preventing existential risks—so working to reduce these risks becomes the highest priority.

Incidentally, the flaw in this theory is in thinking you understand what all the existential risks are.

Suppose you clock "malicious AI" as a huge risk and then hamper AI, but it turns out the bigger risk is not doing space exploration, which AI would have accelerated, because something catastrophic yet already-inevitable is going to happen to the Earth in a few hundred years and if we're not sustainably multi-planetary by then it's all over.

The thing evolution teaches us is that diversity is a group survival trait. Anybody insisting "nobody anywhere should do X" is more likely to cause an ELE than prevent one.

[TeMPOraL]

> Incidentally, the flaw in this theory is in thinking you understand what all the existential risks are.

Rationalist community understands that very well. They even know how to put bounds on the unknowns and their own lack of information.

> The thing evolution teaches us is that diversity is a group survival trait. Anybody insisting "nobody anywhere should do X" is more likely to cause an ELE than prevent one.

Right. Good thing they'd agree with you 100% on this.

[astrange]

> They even know how to put bounds on the unknowns and their own lack of information.

No they don't. They think they can do this because they've accidentally reinvented the philosophy "logical positivism", which philosophers gave up on because it doesn't work. (This is similar to how they accidentally reinvented reconstructing arguments and called it "steelmanning".)

https://metarationality.com/probability-limitations

[TheAncientGeek]

https://www.greaterwrong.com/posts/dTkWWhQkgxePxbtPE/no-logi...

[AnthonyMouse]

The nature of unknowns is that you don't know them.

What's the probability of AI singularity? It has never happened before so you have no priors and any number you assign will be pure speculation.

[TeMPOraL]

Same is true about anything you're trying to forecast, by definition of it being in the future. And yet people have figured out how to make predictions more narrow than shrugging.

[jandrese]

"It is difficult to make predictions, especially about the future."

Most of the time we make predictions based on how similar events happened in the past. For completely novel situations it's close to impossible to make a prediction and reckless to base policy on such a prediction.

[ffsm8]

That's strictly true, but I feel like you're misunderstanding something. Most people aren't actually doing anything truly novel, hence very few people ever actually have to even attempt to predict things in this way.

But it was necessary at the beginning of flight and the flight to the moon would've never been possible either without a few talented people being able to make predictions about scenarios they knew little about.

There are just way too many people around nowadays, which is why most of us never get confronted with such novel topics and consequently we don't know how to reason about it

[senko]

>> It has never happened before

> Same is true about anything you're trying to forecast, by definition of it being in the future

There might be some flaws in this line of reasoning...

[TeMPOraL]

Yes, but making it more nuanced still doesn't change the point.

Singularity obviously never happened before, and if anyone bothered to read up on what they're talking about, they'd realize that no one is trying to predict what happens then, because the singularity is defined as the time when changes accelerate to such a degree that we have no baseline to make any predictions whatsoever.

So when people speculate on when that is, they're trying to forecast the point forecasting breaks; they do it by extrapolating from known examples and trends, to which we do have baselines.

Or, in short: we know how it is to ride an exponent, we just never rode one long enough to fall off of it; predicting singularity is predicting when the exponent gets steep enough we can't follow, which is not unlike predicting any other trend people do. Same methods and caveats apply.

[freejazz]

"And the general absolutist tone of the community. The people involved all seem very... Full of themselves ?"

>And yet people have figured out how to make predictions more narrow than shrugging

And?

[bayarearefugee]

That's only one flaw in the theory.

There are others, such as the unproven, narcissistic and frankly unlikely-to-be-true assumption that humanity continuing to exist is a net positive in the long run.

[adastra22]

"net positive" requires a human being existing to judge.

[bayarearefugee]

> "net positive" requires a human being existing to judge.

This is effectively a religious belief you are espousing.

[adastra22]

What do you mean? Religion seems like a total nonsequitur here. If anything I would argue the opposite: you would need religion or something like it for moral statements to have any meaning without the existence of sentient beings.

[christophilus]

A net positive for whom?

[bayarearefugee]

Everything. of which humanity is a miniscule part.