I can't possibly know the actual problem. It could be a multitude of issues, hell it could even be one unofficial babel plugin, but it is absolutely not babel taking 20s.
Babel is essentially `cp` but with a slight twist: AST -> AST conversion. That doesn't take a huge amount of time.
This is purely anecdotal, and we could very well have done something wrong with our configuration, but I had my build iteration times cut in more than half after switching our project to CommonJS + ES6 Typescript from es2015-preset Babel.
I think we were using Gulp without any hot module replacement, so there are other ways we could have sped up our build time, but Typescript was good enough.