[odnes]

But Tesla's fleet is getting more driving experience every day than you will get in your entire life, and every time they train on one of those exceptions, the entire fleet will benefit.

Humans can reason about things that haven't to them happened before. Today's machine learning systems cannot. As you say, to react appropriately they must have been trained to do so using human annotated data.

The argument against FSD is that you would need an infinite number of annotated examples, and an infinite number of subroutines for behaving in any identified situations, because the space of driving is effectively infinite.

Until machines are able to do general reasoning about things they've not experienced before then FSD is not happening. By the way, Demis Hassabis thinks that this sort of transfer learning is the key to solving AI.