I think you could have less steps here. Why not have the person sending out the initial invite pick all the times possible for them, and just let the collaborators choose the ones that work for them, and then send out an email that works for the most people.
I think we might have some bad UX here, you can actually propose multiple times in the wizard (as a creator). We allow other people to propose a time in case none of the original times work for them. We just deployed some stuff that will hopefully make this more clear.
I'm in the market for a solution here, but really need one with calendar integration. If I have to go back and forth between the app and my calendar it doesn't save me that much time over email.