That's the sad truth for a lot of modern compiled languages (I think Go is an exception). While I did not dig into their compiler internals, it just seems an inevitable consequence from a powerful language (Go is pretty simple in that regard). C++, Rust, Scala, Kotlin, Swift, they all have significantly longer compilation times compared to their predecessors. Probably that's the price we have to pay.
Chez compiles a 80Kloc project of mine in less than a second, and apparently that is still 2x slower than before they switched to a nanopass compiler.
There is no reason compiling things should not compile at similar speeds to chez unless you are telling your compiler to ootimise everything.
C++/ruae are just exceptionally slow and it became the new bottom line somehow. I know rust does a lot of housekeeping, but after using chez and sbcl, 45s for a 1.5kloc project is killing me inside.