[CrendKing]

Why can't Cargo have a system like PyPI where library author uploads compiled binary (even with their specific flags) for each rust version/platform combination, and if said binary is missing for certain combination, fallback to local compile? Imagine `cargo publish` handle the compile+upload task, and crates.io be changed to also host binaries.

[cesarb]

> Why can't Cargo have a system like PyPI where library author uploads compiled binary

Unless you have perfect reproducible builds, this is a security nightmare. Source code can be reviewed (and there are even projects to share databases of already reviewed Rust crates; IIRC, both Mozilla and Google have public repositories with their lists), but it's much harder to review a binary, unless you can reproducibly recreate it from the corresponding source code.

[woodruffw]

I don’t think it’s that much of a security nightmare: the basic trust assumption that people make about the packaging ecosystem (that they trust their upstreams) remains the same whether they pull source or binaries.

I think the bigger issues are probably stability and size: no stable ABI combined with Rust’s current release cadence means that every package would essentially need to be rebuilt every six weeks. That’s a lot of churn and a lot of extra index space.

[aloha2436]

> remains the same whether they pull source or binaries.

I don't think that's exactly true, it's definitely _easier_ to sneak something into a binary without people noticing than it is to sneak it into rust source, but there hasn't been an underhanded rust competition for a while so I guess it's hard to be objective about that.

[littlestymaar]

Pretty much nobody does those two things at the same time:

- pulling dependencies with cargo - auditing the source code of the dependencies they're building

You are either censoring and vetting everything or you're using dependencies from crates.io (ideally after you've done your due diligence on the crate), but should crates.io be compromised and inject malware in the crates' payload, I'm ready to bet nobody would notice for a long time.

I fully agree with GP that binary vs source code wouldn't change anything in practice.

[ninkendo]

> Pretty much nobody does those two things at the same time: - pulling dependencies with cargo - auditing the source code of the dependencies they're building

Your “pretty much” is probably weaseling you out of any criticism here, but I fully disagree:

My IDE (rustrover) has “follow symbol” support, like every other IDE out there, and I regularly drill into code I’m calling in external crates. Like, just as often as my own code. I can’t imagine any other way of working: it’s important to read code you’re calling to understand it, regardless of whether it’s code made by someone else in the company, or someone else in the world.

My IDE’s search function shows all code from all crates in my dependencies. With everything equal regardless of whether it’s in my repo or not. It just subtly shades the external dependencies a slightly different color. I regularly look at a trait I need from another crate, and find implementations across my workspace and dependencies, including other crates and impls within the defining crate. Yes, this info is available on docs.rs but it’s 1000x easier to stay within my IDE, and the code itself is available right there inline, which is way more valuable than docs alone.

I think it’s insane to not read code you depend on.

Does this mean I’m “vetting” all the code I depend on? Of course not. But I’m regularly reading large chunks of it. And I suspect a large chunk of people work the way I do; There are a lot of eyeballs on public crates due to them being distributed as source, and this absolutely has a tangible impact on supply chain attacks.

[littlestymaar]

You answer your own argument here:

> Does this mean I’m “vetting” all the code I depend on? Of course not.

Inspecting public facing parts of the code is one thing, finding nasty stuff obfuscated in a macro definition or in a Default or Debug implementation of a private type that nobody is ever going to check outside of auditors is a totally different thing.

> My IDE (rustrover) has “follow symbol” support

I don't know exactly how it works for RustRover, since I know Jetbrain has reimplemented some stuff on their own, but if it evaluates proc macros (like rust-analyzer) does, then by the time you step into the code it's too late, proc macros aren't sandboxed in any ways and your computer could be compromised already.

[fc417fc802]

If you have reproducible builds it's no different. Without those binaries are a nightmare in that you can't easily link a given binary back to a given source snapshot. Deciding to trust my upstream is all well and good but if it's literally impossible to audit them that's not a good situation to be in.

[woodruffw]

I think it’s already probably a mistake to think that a source distribution consistently references a unique upstream source repository state; I don't believe the crate distribution layout guarantees this.

(I agree that source is easier to review and establish trust in; the observation is that once you read the upstream source you’re in the same state regarding distributors, since build and source distributions both modify the source layout.)

[littlestymaar]

No stable ABI doesn't mean the ABI changes at every release though.

[hobofan]

It might as well. If there is no definition of an ABI, nobody is going to build the tooling and infrastructure to detect ABI compatibility between releases and leverage that for the off-chance that e.g. 2 out of 10 successive Rust releases are ABI compatible.

[littlestymaar]

Why wouldn't they do exactly that if they decided to publish binary crates…

Nobody does that right now because there's no need for that, but it doesn't mean that it's impossible in any way.

Stable ABI is a massive commitment that has long lasting implications, but you don't need that to be able to have binary dependencies.

[zozbot234]

You can have binary dependencies with a stable ABI; they're called C-compatible shared libs, provided by your system package manager. And Cargo can host *-sys packages that define Rust bindings to these shared libs. Yes, you give up on memory safety across modules, but that's what things like the WASM Components proposals are for. It's a whole other issue that has very little to do with ensuring safety within a single build.

[pjmlp]

Yet other ecosystems handle it just fine, regardless of security concerns, by having signed artifacts and configurable hosting as an option.

[nicoburns]

> Unless you have perfect reproducible builds

Or a trusted build server doing the builds. There is a build-bot building almost every Rust crate already for docs.rs.

[majewsky]

docs.rs is just barely viable because it only has to build crates once (for one set of features, one target platform etc.).

What you propose would 1) have to build each create for at least the 8 Tier 1 targets, if not also the 91 Tier 2 targets. That would be either 8 or 99 binaries already.

Then consider that it's difficult to anticipate which feature combinations a user will need. For example, the tokio crate has 14 features [1]. Any combination of 14 different features gives 2^14 = 16384 possible configurations that would all need to be built. Now to be fair, these feature choices are not completely independent, e.g. the "full" feature selects a bunch of other features. Taking these options out, I'm guessing that we will end up with (ballpark) 5000 reasonable configurations. Multiply that by the number of build targets, and we will need to build either 40000 (Tier 1 only) or 495000 binaries for just this one crate.

Now consider on top that the interface of dependency crates can change between versions, so the tokio crate would either have to pin exact dependency versions (which would be DLL hell and therefore version locking is not commonly used for Rust libraries) or otherwise we need to build the tokio crate separately for each dependency version change that is ABI-incompatible somewhere. But even without that, storing tens of thousands of compiled variants is very clearly untenable.

Rust has very clearly chosen the path of "pay only for what you use", which is why all these library features exist in the first place. But because they do, offering prebuilt artifacts is not viable at scale.

[1] https://github.com/tokio-rs/tokio/blob/master/tokio/Cargo.to...

[nicoburns]

You could get a lot of benefit from a much smaller subset. For example, just the "syn" crate with all features enabled on tier 1 targets (so ~8 builds total) would probably save a decent chunk off almost everybody's build.

[epage]

It runs counter to Cargos curreat model where the top-level workspace has complete control over compilation, including dependencies and compiler flags. I've been floating an idea of "opaque dependencies" that are like python depending on C libraries or a C++ library dependening on a dynamic library.

[littlestymaar]

That would work for debug builds (and that's something that I would appreciate) but not for release, as most of the time you want to compile for the exact CPU you're targeting not just for say “x86 Linux” to make sure your code is optimized properly using SIMD instructions.

[surajrmal]

A trustworthy distributed cache would also work very well for this in practice. Cargo works with sccache. Using bazel + rbe can work even better.