Less safe is up to the statistics. My guess is that its safe "enough" right now and only gets safer beyond humans. But the other point is more interesting: Flexibility. Since you don't need a driver, you can have a different kind of fleet management that - yes, I am an optimist - would be more available 24/7 than with human drivers.
See that's the thing, I don't see how it can ever be safer than all but the least competent human drivers.
I have yet to see an autonomous vehicle that could cope with anything other than a completely unobstructed road with everyone behaving themselves, and I can't see a way for autonomous vehicles to progress beyond that.
If you don't think taxis are available 24/7, you haven't met anyone that drives a taxi. Someone is always up and about.