I thought that Uber drivers could not see your destination until they picked you up. Are drivers taking off and refusing at the moment of pickup in SF? Seems like that would be pretty easy to catch behavior.
In NYC, I had several drivers refuse to let me into the car until I told them where I was going. I never took those cars and made them cancel it (and then reported it). However, it is common practice.