"Trusting the Rust org not to screw you" is one part, another part is trusting the Rust server operators to defend against server compromise by any third party. So trusting the intentions is not sufficient.
The same thing applies to any binaries downloaded from their site, so unless you you've got signed binaries (that use an independently obtained/verified chain of trust), trusting the server is your your only option. Even with signed binaries, you're still trusting the entity that holds the signing key.
In real world trust is not so binary. In a risk assessment I'd be interested evaluating the level of assurance there is in the supply chain of how you get your binaries and artifacts. Some of it can be done using crypto like you say, some of it could be eg published audit reports from a reputable evaluator or other credible information about the processes.