[yshui]

This is pretty dumb on Debian's part. First of all I don't understand why they insist crate dependencies must be pulled from their repository. They are just source code, not built binary. AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that. Arch does this, Gentoo does this, NixOS does this, why does Debian has to be different?

Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too.

This is just all-around weird what Debian is doing.

(Full disclosure, I am the one who introduced the first piece of Rust code into bcachefs-tools)

[acka]

Debian has a Social Contract[1] as well as guidelines {the DFSG}[2] regarding the commitment to only distribute free and open source software that all package maintainers must adhere to. This means that package maintainers must check the licenses of source code and documentation files, clear up any ambiguities by talking to upstream, and (as a last resort) even excise code and/or documentation from Debian's copy of the codebase if it doesn't meet the requirements of the DFSG.

In practise, this means that Debian has to make its own copy of the source code available from a Debian-controlled repository, to ensure that no (accidental or otherwise) change to an upstream source archive can cause non-DFSG compliant Debian source or binary packages to be distributed.

[1] https://www.debian.org/social_contract

[2] https://wiki.debian.org/DebianFreeSoftwareGuidelines

[koverstreet]

That justification doesn't work here - I provide vendored tarballs, so the source code availability argument is moot.

[righthand]

But it’s not guaranteed. The Debian way provides a method of allocating responsibility. So if anything does go wrong they can point to a responsible party, the package maintainer. By providing tarball source you’re trying to placate responsibility of some code. You could build those tarballs on a different machine/different OS and any issues wouldn’t technically be your problem because “it’s just deps”.

[ouEight12]

> Arch does this, Gentoo does this, NixOS does this, why does Debian has to be different?

I say this as someone who ran Gentoo for years and daily drives Arch today.

Because sometimes you don't want entire swaths of your server being rebuilt/tinkered with on a regular basis under the hood. "Move fast, break everything" is great in dev/test land, or a prod environment where the entire fleet is just containers treated like cattle, but contrary to what the SREs of the valley would believe, there's a whole ecosystem of 'stuff' out there that will never be containerized, where servers are still treated like pets, or rather, at least "cherished mules", that just do their job 24/7 and get the occasional required security updates/patches and then go right back to operating the same way they did last year.

[cesarb]

> AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that.

AFAIK, most traditional distributions do that, not just Debian. They consider it important that software can be rebuilt, even in the far future, with nothing more than a copy of the distribution's binary and source packages. Doing anything which depends on network access during a build of the software is verboten (and AFAIK the automated build hosts block the network to enforce that requirement).

Keep also in mind that these distributions are from before our current hyper-connected time; it was common for a computer to be offline most of the time, and only dial up to the Internet when necessary. You can still download full CD or DVD sets containing all of the Debian binaries and source code, and these should be enough to rebuild anything from that distribution, even on an air-gaped computer.

> Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too.

That is often done for C libraries; for instance, Debian stable has both libncurses5 and libncurses6 packages. But it's a lot of work, since for technical reasons, each version has to be an independent package with a separate name, and at least for Debian, each new package has to be reviewed by the small ftpmaster team before being added to the distribution. I don't know whether there's anything Rust-specific that makes this harder (for C libraries, the filenames within the library packages are different, and the -dev packages with the headers conflict with each other so only one can be installed at a time).

There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on).

[kijin]

> There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on).

This is the most important part. Debian LTS maintains packages for 5 years. Canonical takes Debian sources, and offers to maintain their LTS for 10 years. Red Hat also promises 10 years of support. They don't want anything in the core part of their stable branches that they can't promise to maintain for the next 5-10 years, when they have no assurance that upstream will even exist that long.

If you want to move fast and break things, that's also fine. Just build and distribute your own .deb or .rpm. No need to bother distro maintainers who are already doing so much thankless work.

[zamadatix]

I didn't get the impression anyone bothered the maintainers to do this, the article implies they chose to package it on their own accord.

[kijin]

Yes, and they chose to stop packaging it, again on their own accord.

Except this time, they seem to have made the decision because upstream was bothering them too much.

[zamadatix]

No arguments there, I'm more talking what that means for the "build and distribute your own dev/rpm" part that follows. Why are the only options "do work for maintainers" or "provide a prebuilt package for the distro", what happened to "nobody said this needed to be done yet"?

[kijin]

No problem, if upstream doesn't want their software packaged for Distro X, nobody needs to do anything.

The thing about Linux filesystems, though, is that they consist of two parts: the kernel patch and the userspace tooling. Bcachefs is already in the kernel, so it's a bit awkward to leave out bcachefs-tools. Which is probably why it got packaged in the first place. Stable distros generally don't want loose ends flailing about in such a critical part of their system. If nobody wants to maintain bcachefs-tools for Debian, Debian will probably remove bcachefs entirely from their kernel as well.

[jonhohle]

I’d consider the issue to be the opposite. Why does every programming language now have a package manager and all of the infrastructure around package management rather than rely on the OS package manager? As a user I have to deal with apt, ports, pkg, opkg, ipkg, yum, flatpak, snap, docker, cpan, ctan, gems, pip, go modules, cargo, npm, swift packages, etc., etc., which all have different opinions of how and where to package files.

On packaged operating systems (Debian, FreeBSD) - you have the system’s package manager to deal with (apt, pkg respectively). I can have an offline snapshot of _all_ packages that can be mirrors from one place.

IMHO, every programming language having its own package system is the weird thing.

[MobiusHorizons]

If you are a developer you almost always eventually need some dependencies that don’t ship with the os package manager, and once some of your dependencies are upstream source, you very quickly find that some dependencies of the sources you download rely on features from newer versions of libraries. If you have multiple clients, you may also need to support both old and new versions of the same dependencies depending on who the work is for. Package managers for a Linux distribution have incompatible goals to these (except maybe nix)

[epage]

We want to make our software available to any system without every library maintainer being a packaging expert in every system.

The user experience is much better when working within tkese packaging systems.

You can control versions of software independent of the machine (or what distros ship).

Or in other words, the needs of software development and software distribution are different. You can squint and see similarities but the fill different roles.

[jonhohle]

So every user has to be an expert in every package manager instead? Makes sense. Make life easy for the developer and pass the pain on to thousands of users. 20 years ago you may or may not support RPM and DEB and for everyone else a tarball with a make file that respected PREFIX was enough. (Obviously a tarball doesn’t support dependencies.)

[epage]

Why would users need to be an expert in them?

[SkiFire13]

Because OS packaging stuff sucks. It adds an enourmous barrier to sharing and publishing stuff.

Imagine that I make a simple OS-agnostic library in some programming language and want to publish it to allow others to use it. Do I need to package for every possible distro? That's a lot of work, and might still not cover everyone. And consider that I might not even use Linux!

A programming language will never get successful if that is what it takes to built up a community.

Moreover in the case of Rust distos are not even forced to build using crates.io. However the downside is that they have to package every single dependency version required, which due to the simplicity of publishing and updating them have become quite a lot and change much often than they would like.

The funny thing is that in the C/C++ world it's common to reimplement functionality due to the difficulty of using some dependencies for them. The result is not really different from vendoring dependencies, except for the reduced testing of those components, and this is completly acceptable to distros compared to vendoring. It makes no sense!

[pornel]

1. Because Windows/macOS/iOS/Android don't have a built-in package manager at the same granularity of individual libraries, but modern programming languages still want to have first-class support for all these OSes, not just smugly tell users their OS is inferior.

2. Because most Linux distros can only handle very primitive updates based on simple file overwrites, and keep calling everything impossible to secure if it can't be split and patched within limitations of C-oriented dynamic linker.

3. Because Linux distros have a very wide spread of library versions they support, and they often make arbitrary choices on which versions and which features are allowed, which is a burden for programmers who can't simply pick a library and use it, and need to deal with extra compatibility matrix of outdated buggy versions and disabled features.

From developer perspective with lang-specific packages

• Use 1 or 2 languages in the project, and only need to deal a couple of package repositories, which give the exact deps they want, and it works the same on every OS, including cross-compilation to mobile.

From developer perspective of using OS package managers:

• Different names of packages on each Linux distro, installed differently with different commands. There's no way to specify deps in a universal way. Each distro has a range of LTS/stable/testing flavors, each with a different version of library. Debian has super old useless versions that are so old it's worse than not having them, plus bugs reintroduced by removal of vendored patches.

• macOS users may not have any package manager, may have an obscure one, and even if they have the popular Homebrew, there's no guarantee they have the libs you want installed and kept up-to-date. pkg-config will give you temporary paths to precise library version, and unless you work around that, your binary will break when the lib is updated.

• Windows users are screwed. There are several fragmented package managers, which almost nobody has installed. They have few packages, and there's a lot of fiddly work required to make anything build and install properly.

• Supporting mobile platforms means cross-compilation, and you can't use your OS's package manager.

OS-level packaging suuuuuuuucks. When people say that dependency management in C and C++ is a nightmare, they mean the OS-level package managers are a nightmare.

[trueismywork]

Programming language package managers are more for development than deployment.

[jonhohle]

Yet as a user who programs only in a handful of those languages, I’ve used them all and continue to come across new ones.

[Twirrim]

>AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that.

Note your examples are all bleeding edge / rolling distributions. Debian and the non-bleeding edge distributions go a different route and focus on reproduce-ability and security, among other things.

With the "get from crates.io" route, if someone compromises/hijacks a crate upstream, you're in trouble, immediately. By requiring vendoring of sources, that requires at least some level of manual actions by maintainers to get that compromised source in to the debian repositories to then be distributed out to the users. As you get in towards distributions like RHEL, they get even more cautious on this front.

[doublepg23]

NixOS is not a rolling distro.

[Palomides]

debian is substantially older than all of those distros, and you named three that happen to, in my view, been designed specifically in reaction against the debian style of maintenance (creating an harmonious, stable set of packages that's less vulnerable to upstream changes), so it's strange to say that debian is the odd one out

keeping a debian-hosted copy of all source used to build packages seems like a very reasonable defensive move to minimize external infrastructure dependencies and is key in the reproducible builds process

there's definitely a conflict with the modern style of volatile language-specific package management, and I don't think debian's approach is ideal, but there's a reason so many people use debian as a base system

also it seems like the idea of maintaining stable branches of software has fallen out of vogue in general

[wojciii]

Perhaps they want to have builds they can reproduce years later?

When libraries are removed you have to update your build. This will not be the case if you use your own infrastructure (crates.io mirror).

[cozzyd]

Part of a distros job is to audit all the packages (even if just to a minimal extent), and in many cases patch for various reasons. This is much harder if the source is external and there are N copies of everything.

[tsimionescu]

Because Debian and similar distros have a goal of maintaining all of the software that users are expected to use. And this means they commit to fixing security issues in every single piece of software they distribute.

A consequence of this is that they need to fix security issues in every library version that any application they distribute uses, including any statically-linked library. So, if they allow 30 applications to each have their own version of the same library, even Rust crates, and those versions all have a security issue, then the Debian team needs to find or patch themselves 30 different pieces of code. If instead they make sure that all 30 of those applications use the same version of a Rust crate, then they only need to patch one version. Maybe it's not 30 times less work, but it's definitely 10 times work. At the size of the Debian repos, this is an extremely significant difference.

Now, it could be that this commitment from Debian is foolish and should be done away with. I certainly don't think my OS vendor should be the one that maintains all the apps I use - I don't even understand the attraction of that. I do want the OS maintainer to handle the packaging of the base system components, and patch those as needed for every supported version and so on - and so I understand this requirement for them. And I would view bcachefs-tools as a base system component, so this requirement seems sane for it.

[Brian_K_White]

It's pretty dumb (your words if you don't like them) not to understand (your words if you don't like them) that, how, and why, different distributions "has to be different". Debian is not Arch or Nix. A tractor is not a race car is not a submarine, even though they are all vehicles.

This seems like a very basic concept for anyone purporting to do any sort of engineering or designing in any field.

If it were me, I would not be so eager to advertize my failure to understsnd such basics, let alone presume to call anyone else pretty dumb.