Rust packages tend to have large dependency graphs which doesn't help the compile times. I've read the serde is a bit of a hog in that respect, for example. The rust philosophy is for a large language and large library too.
I highly recommend checking out makepad [1] - they have +100k of rust code and the compile time is around 10-15 seconds on commodity hardware.
However they are obsessed about performance. They reason for such speedy compile times, like you say, is that makepad has almost no external dependencies.
[1] https://github.com/makepad/makepad/