When you can write a busy beaver machine in the type system, LOC ceases to be a good indicator or how long something should take to typecheck, imo. If you're frustrated with your build, you should use the trace tools on the TS wiki[1] to track down what types are slow to check, so you can attribute the slowness to the appropriate library authors/yourself and decide for yourself if the speed/correctness tradeoff they've made is right for you.
Personally, I find that I don't really need full type checking on every build. For most small changes, the errors shown in vscode are sufficient while editing, and then full type checking can be run occasionally when needed (and in CI of course).
Your IDE can be doing type checks on whatever file(s) you're working on, you can use esbuild or swc to compile to javascript to make sure it runs correctly, and you can periodically use tsc to fully compile and typecheck your entire codebase to catch anything that you somehow missed in the IDE.
[1]https://github.com/microsoft/TypeScript-wiki/blob/main/Perfo...