They are not afraid to operate their own vehicles. They are afraid you will kill them.
You just said that you do not care how many people you kill - regardless of whether they are pedestrians, whether they are driving cars or whether they are on the bus. That is what people react to.
No, I mean that they are not prioritizing you and many make poor choices.
Replacing bad other drivers with good autonomous systems is likely a great trade off for you, even if you are in an autonomous vehicle that is eager to sacrifice you if there is an unavoidable incident.
As for me I actually like driving and I'm good at it. I'm not afraid of operating my own vehicle like so many people seem to be