I don't even bother with vanilla JS. Babel and Typescript let you magically use the latest features without caring about compatibility at all really, except CSS
People often promote this but there’s a huge gap between script tag to include jquerg.min (for example) and requiring the entire node/npm ecosystem to cross compile etc everything else (provided you weren’t already). This is a high tax to pay.
Typescript is super easy to bring into a project. Just rename .js to .ts and turn the validations off. Any valid JS is also valid TS. Build step is one command to run tsc. You can turn things on slowly as you refactor into idiomatic typescript.
React is a PITA to setup and learn, as is Angular. They're the modern equivalent of Java EE. "Heavy" but if you have something complex to do and know your way around they make hard things easy. The learning curve is tough but so worth it
Transpiled JS specifically. JS is the only popular language with a great deal of differences between versions and runtimes, mostly due to rapid improvements. If you use "vanilla" ES2017 without transpilation you're in for a deluge of bug reports.
Instead of worrying about browser Y supporting feature X, I just ram everything through Babel or Typescript compilers and use all the newest features.
We haven't had a single browser specific JS bug since we started using Typescript on ny project last year. And async is so damn nice.
Babel's output is a bit obtuse sometimes, but Typescript was designed to output idiomatic JS, so even with no sourcemaps code is perfectly readable before minification
In my experience (compiling TS + babel down to ES5), the compiled output is generally fine. Unless you're trying to squeeze every last kb out of it, asset optimization and NPM dependency bloat are much bigger optimization opportunities anyway.
Haven't looked at Babel's, but the output from Typescript is human readable and not much bigger than the original. The more polyfills (lower your target lib aka ES3 vs ES2015) The bigger the output.
Even targeting the oldest crappiest browsers doesn't add that much. Some maybe 50kb for polyfills injected and code bloat of 30% ish. Not much price to pay to use the latest features without caring about compatibility