[aseerdbnarng]

I price motor insurance for a living and in my models the absolute worst 'insurable' risks (ones that pass underwriting scrutiny) will have some sort of impact every 3rd year. If we filter that to more severe cases where there's bodily injury it moves out to 8/9 years.

What this means is the absolute worst drivers will go ~100k miles between causing serious accidents so the threshold between good, bad and awful drivers is really far out into the tails of the distribution. Without actual driving data to work from we don't know how safe FSD is and youtube videos are purposefully selected for views. Even so the sheer number and 'randomness' of fails is concerning because nothing like alcohol, speeding or inexperience to point to.

[giraffe_lady]

Yes thank you. I was literally thinking "the worst driver I know has been in two accidents in the 20 years of our friendship, that must be pretty close to 99.9% safe."

I don't have very strong intuition about risks this small. I do know the thing that ultimately lead me to being able to beat nethack at will was internalizing the idea that "something that's 99% safe is almost certain to kill you over the course of a run." Feels similar.

[pxmpxm]

This, as someone in tangentially related industry I see the very same issue - you need a LOT of tail events to be able to train the models on them and obviously tail events are rare by construction. So it will be all smooth sailing until you hit the 1/1e9 and kill someone with your model3.