Having used Scala for some years with it's abysmal compile speed, I fear it has to do with complexity of the language and features, it was "implicit" in Scala, it might be the borrow checker in Rust (IANACG)
Note that I left out Scala, while I included other languages of similar complexity, exactly because improving compile times has never been a major focus in Scala land, versus features.
Example, development effort wasted on Scala 3 Python like syntax.
There are several performance reports that show that the major culprit is the pile of LLVM IR sent to the backend.
The largest single bit tends to be codegen, which is why cargo check is so much faster than cargo build.