Crates is especially challenging -- the build process is very expensive on all resources (cpu, ram, and disk), and packages are very hard (impossible?) to cache. It works super well on the Pro plan.
Have you tried cargo-binstall instead? They're pre-compiled crate binaries that fall back to cargo install, works very well. Also, you might want to look into the mold linker as well as sccache.