Author here. This is a good point. In this case we found it does slow down clean builds somewhat since we're pulling down + compiling Rust dependencies now as well as Elixir. While actually editing Elixir code I haven't experienced any noticeable slowdown for incremental builds - the Rust code is mostly static.
It's one reason I prefer lighter languages for NIF's. My personal preference is to use Nim and Nimler [1]. It generally compiles quicker than Rust while providing most of the same benefits. To be fair compiling a small Rust library doesn't take too much time. Especially compared to the performance numbers from those Rust NIFs!