The down side is that not only is all of your data on your movements tracked, but it is also being sold to others, including the government. Another nail in privacy's coffin.
Anonymous vehicular travel is dead or soon-to-be-dead anyway. Police are routinely using license plate scanners for parking enforcement and for checking vehicle alert status. Toll facilities are using license plate scanning to bill you for congestion fees. Smart phone apps report your whereabouts sometimes by choice, sometimes not.
This isn't to say that pervasive tracking of individuals' whereabouts isn't a problem. I just think that self-driving cars aren't going to make the situation any worse than it is / will already be (exact timing is modulo your present jurisdiction).
This isn't to say that pervasive tracking of individuals' whereabouts isn't a problem. I just think that self-driving cars aren't going to make the situation any worse than it is / will already be (exact timing is modulo your present jurisdiction).