Overcast (a podcasting app) has great features to optimize the high speed listening experience. They have variable speed, a great silence trimmer, and a voice boost that makes speech clearer.
I just wish I could listen to arbitrary audio with Overcast. As it is I have a blog set up to feed Overcast audio that I give it, but it feels super clunky.