I can't possibly know the actual problem. It could be a multitude of issues, hell it could even be one unofficial babel plugin, but it is absolutely not babel taking 20s.
Babel is essentially `cp` but with a slight twist: AST -> AST conversion. That doesn't take a huge amount of time.
This is purely anecdotal, and we could very well have done something wrong with our configuration, but I had my build iteration times cut in more than half after switching our project to CommonJS + ES6 Typescript from es2015-preset Babel.
I think we were using Gulp without any hot module replacement, so there are other ways we could have sped up our build time, but Typescript was good enough.
Babel is essentially `cp` but with a slight twist: AST -> AST conversion. That doesn't take a huge amount of time.