I don't see how the situation is similar. The robotaxi is parked in front of you, you connect to it and move it away. The entire context is right there.
well honestly, if we're saying that the person being inconvenienced is the one who gets to take control, there's probably gonna be a lot of angry people driving these cars off of bridges or into ditches or something. Giving an incredibly frustrated person control of a separate car, though better for context, is probably worse for desired outcome.
How about the robotaxi company driving in respect to regulations and other's property, then we wouldn't have this discussion. Oh, it's technically not possible? Well, then don't drive them.