I tried changing various defaults but lots of them seemed to no longer work. Maybe I only tried the options to increase the speed or I didn’t restart enough things or I spelled something wrong. I also tried looking at the binaries for relevant-looking names of defaults to change and didn’t find any (including the options that old internet advice recommended changing)
The keyboard shortcuts to do this animate much quicker than the swipe does. In the past I've set up BetterTouchTool to press the corresponding key on left and right swipes.
The only disadvantage is that there is no way to "peek" the previous desktop like you can with the built in swipes.