[Sanddancer]

Will this play nicely with the package management tools OSes already have, or is this going to end up being yet another source for files/packages to accumulate that are outside the view of well-documented and designed administrative tools?

[alextgordon]

Ha! There is nothing well-designed about the mutually incompatible, political hell that is OS package managers.

The best solution (as taken by npm, virtualenv and others) is to install libraries locally to the project that is building them.

That way, package management becomes the sole concern of the build system.

"Accumulation" is a good thing, it means each project has the exact version of a package that it was tested with, not some later version that a sysadmin decides it "probably should work with".

[bryanlarsen]

OS package managers make two assumptions:

- that packages follow semver

- that the OS packagers are in a better position to test package combinations.

If the author releases a new version of libfoo, and A, B and C in an OS repo depend on libfoo, then the OS packagers do not release a new version of libfoo until the tests for A, B & C pass.

These are two good assumptions, and the language package world would be in much better shape if they followed those assumptions too.

[pjmlp]

Which OS are "OS package managers"? There is a world out there greater than GNU/Linux.

[steveklabnik]

Cargo assumes the former.

[Sanddancer]

Which is why we have testing and staging environments to make these changes on first. Conversely, when everything is pulling in their own version of a library, you run into situations like the zlib buffer overflow, where you had huge numbers of programs that needed to be rebuilt because no one used system-packaged libraries. Obsolete and vulnerable software is a liability, and not having tools which can readily tell what software needs to be updated makes quite a few peoples' jobs quite a bit harder.

[shadowmint]

You can't have it all.

Either you rely on system librarys OR every binary / library pulls in its own copy of its dependencies (npm model).

The latter lets you have multiple versions of the same library for different dependencies, without confusing your system package manager.

The former might mean 'less build time', but it's pretty much whats wrong with the C/C++ ecosystem; you can't track the system libraries from the package tool, and you get out of sync.

Pillow and freetype specifically pop to mind as an irritatingly broken recent example; when freetype upgrades, perfectly working python projects suddenly stop working because the base system freetype library has been upgraded; and the pinned python libraries that depended on the previous freetype version no longer work, because they rely on the system package manager not to do stupid things.

It would be nice if you could point cargo as a 'local binary cache' to speed things up, and make them work even if the central repository goes down; and that could be package manager friendly, I imagine.

[Sanddancer]

I'd say look at the BSD ports system for an example of a system done well, and is extensible to such issues. If your app depends on projects with irresponsible developers that make unannounced API/ABI changes and the like, the general format of the ports system, and tools for maintaining private repositories like portshaker and poudriere, make it easy to create ports that let you have the best of both worlds. To add to this, one can use metapackages and the like which don't have any files in and of themselves, but will point to the latest version. So, for example, you create a port named libbar, which is always going to get you the latest and greatest, but you also have libbar-1, libbar-12, etc, that you can use when you need an explicit dependency. Additionally, there are ports of the system library -- heimdal, openssl, etc, so you can have a stable base system, but still have more recent versions around for your application. Most of the issues people have with packaging these projects seems to be much more based around inexperience with good packaging tools and practices, rather than the idea of system packaging in and of itself.

[hyperion2010]

Welcome to emerge @preserved-rebuild and revdep-rebuild. Still no better way has been found to date :/.

[sanderjd]

I think it would be just as easy to package rust programs using OS native package management as anything else, and I'm sure packages will be made for popular things for common package managers. But OS native package managers are a royal pain to use when actively developing on a project that has some dependencies, and bespoke Makefiles are a very imperfect solution.

Flipping your question around a bit: will package manager creators and maintainers ever develop better solutions to the use case of development rather than system administration so that we don't have to keep creating these language-specific tools?

[kibwen]

This is an insightful point. Nix is the only package manager I'm aware of that seems like it could fit the bill: https://nixos.org/nix/

[eddyb]

They already have automated Haskell packages from cabal, it shouldn't be hard to integrate cargo with nix once it's more stable.

You wouldn't even require upstream NixOS packages, just place built cargo packages in the nix store, using it like a cache. Then upstream NixOS channels could start accumulating cargo packages, making cargo dependency "builds" faster.

[sanderjd]

Yeah, nix does look pretty awesome and aware of (even actively designed for )both the system administration and development use cases. If it becomes the native package manager for some popular operating systems, I may have to eat my words about language-specific solutions.

[Sanddancer]

Have you taken a look any at the FreeBSD ports system? It uses makefiles to manage pretty much any software install you can think of. Additionally, there's a pretty good infrastructure there for managing custom build trees, etc. Additionally, they also have things like BSDpan that let you use arbitrary Perl modules with the package management system.

[sanderjd]

I've used ports, but I've never developed a project using it, so maybe I should try that out. But "uses makefiles" does not make me particularly optimistic for its pleasantness. I really prefer (and think it's been proven possible to build) declarative, rather than imperative, systems for managing dependencies.

[awj]

Until the various OS package managers stop pretending they are a universe unto themselves, this problem will continue. Languages have to distribute libraries across all of their host systems, so focusing on support for any particular package manager (or small subset thereof) doesn't buy much.

Hopefully the cargo team can come up with a solution that works a little better here, but I wouldn't hold my breath.

[dmacvicar]

Each language having its own tools is fine and I doubt OSes are against that. Those tools just need to support a very specific set of features or scenarios to make packaging easy:

Understand that enterprise OSes are not built on developer Macbooks. Enterprise distros have reproducible builds on automated systems with chroots that contain only what the package needs, no network access and sometimes the build happens inside a virtual machine. Its is almost ridiculous how after maven forgot the topic of being "offline" almost every tool released afterward has done the same mistake.

Understand that Linux distributions sell support and that means being able to rebuild a patched version of a failed package. So whatever dependencies are pinned in Cargo.toml or Gemfile is irrelevant. The OS maker will override them as a result of testing, patching or simply to share one dependency across multiple packages. Distros can't afford having one package per git revision used on a popular distro and then afford to fix the same security issue in all of them.

So having "cargo build" be able to override dependencies and instead look on the installed libraries with a command line switch or env variable helps the packager not having to rewrite the Cargo.toml in the package recipe.

Maven was probably the first tool that made packaging almost impossible and completely ignored the use-case of patching a component of the chain and be able to rebuild it completely.

Semantic versioning is great news, because it allows you to replace the dependencies for a common semantic version (not exactly the same).

For integrating with the C libraries not much needs to be done. If you support pkg-config then you cover 85% of the cases.

[ithkuil]

I had the impression this fills the same role as bower, mpm (non global) dependency or plain custom vendor dirs handled with your favourite git module paraphernalia: your build is the place your dependencies live (+ possible cache directory somewhere else). And, these you use this packaging system to manage your build, not the deployment of your (iirc statically linked) build artifacts.

While technically the cache directory is a place where files can accumulate outside the view of a well documented and designed administrative tools, this is common problem shared with many tools including your favourite browser.