Thanks! BTW you might enjoy setting CARGO_TERM_COLOR=always in your environment :) dtolnay/rust-toolchain does this automatically but it looks like you aren't using that action.
I think in practice you're right that GitHub Actions' caching slowness can end up dominating performance. I don't have any great tips other than to measure.