[k8t]

Let's take a step back and look at the meta-level.

So there's maybe a chance AI is extremely dangerous (as in wipes humanity out), and there's a chance that AI might not be dangerous. There are arguments between how likely each choice is. But the more important fact is that since there's so much at stake (all of humanity), we should likely be really really really sure about things.

For example, let's say you live in a house and you've got your kids and your grandkids and all your lovely pets living here. Let's say there's this mystery device that when you press it, there's a small chance that it could be extremely bad. You'd want to make sure it's OK right? Right?!

I agree that we probably don't need to devote ALL of our resources to make sure it's OK, as we can devote some resources to problems that are actually hurting us right now like diseases, etc. But there are a lot of people in the AI community who think that it may be dangerous. It is rational that we'd want to be absolutely sure that it's safe once we AI happens.

We should take the approach of being DEFAULT cautious when talking about any technological breakthrough that have changes that we cannot reverse.

[idlewords]

The problem with this precautionary reasoning is it leads you to Pascal's Mugging, where you are ready to believe very unlikely things because of the enormous impact they'll have if true.

[k8t]

Yes, I agree with the reasoning behind Pascal's Mugging. But Pascal's Mugging refers to things that have an astronomically low chance of happening — like 0.000000001. But is the chance that AI is dangerous that low? Nobody in the world knows for sure at this point, due to how far away superAI might be, and due to the uncertainties in implementation. Therefore, if we use Bayesian thinking and spread it out, I'm not really sure you could put it below 1% (I pulled this number out of thin air, but everybody at this point is doing the same).

[Analemma_]

There's no magic probability value at which the Pascal's Mugging argument suddenly "kicks in". It's all about the utilitarian tradeoff of, "given this low probability, am I devoting the right amount of resources to preventing this terrible event? Could they be better spent elsewhere? And is the fretting harmful in and of itself, wiping out the expected gain?" The talk is arguing that the answer is yes to both of those latter questions.

[incompatible]

In practice humans don't usually care about such arguments. They'll carry on as usual until something bad actually happens. I'm sure there are plenty of crippled people who regret not wearing a seatbelt, or people with skin cancer who regret not wearing a hat. Not to mention situations on a larger scale like building a lot of nuclear weapons or increasing the C0_2 level in the atmosphere.

[argonaut]

"If there's a 1% chance that Pakistani scientists are helping al-Qaeda build or develop a nuclear weapon, we have to treat it as a certainty in terms of our response. It's not about our analysis ... It's about our response."

- Dick Cheney

[k8t]

I'm completely oblivious to the al-qaeda situation so I don't really know the context of this. But if I try to get at your logic, it would be like saying:

"See? I just went on a drive and didn't get into a crash. All those people that told me to wear a seat-belt were wrong!"

[argonaut]

If you think those two anecdotes are the same, good luck!