[concede_pluto]

This. Make as big a mess as you like on your own box (your sysadmin can pave it when you're hopelessly confused), but everything in prod has to be registered with the one and only package manager because otherwise nobody will know where it's deployed or what its dependencies are or whether they're up to date. cpan/pip/gem/cargo/go get/hackage/melpa are not sysadmin problems.

[ubernostrum]

everything in prod has to be registered with the one and only package manager

No. Never. Not for any reason, ever. Never.

The language's packaging ecosystem and toolchain are:

* Tailored to the language, not the operating system, which means they're reproducible on multiple operating systems. This is important, since your developers are not running RHEL server as their laptop OS, and as a result they'll be using the language toolchain regardless of what your "sysadmin" does to the production environment.

* More likely to be up-to-date and/or update-able than distro-format packages. Unless you want to be running two years ago's version of your libraries (or older), the only way you'll get distro-format packages is to build them yourself... which requires you to go grab them from the language's package system, since that's where they get published, and maintain your own pipeline to re-package them into the distro format. Now you've injected additional moving parts into your systems where none were needed.

Distro packages are only for the base operating system and things like your HTTP daemon. For application code and dependencies, the distro packages should only be involved insofar as they bootstrap you to the point of being able to use the language's toolchain. Insisting on distro-format packages for the whole thing is the path to overcomplex builds/deploys and difficult-to-update codebases.

because otherwise nobody will know where it's deployed or what its dependencies are or whether they're up to date

I can, at a glance, look at any application in production where I work and see what its full dependency tree is and whether those dependencies are up-to-date (and if not, whether they're just outdated or also subject to security advisories). Using things built on the language toolchain. Really. And this is not new cutting-edge technology here, we've had that capability for a good number of years now!

Even better, I can match up that information to what upstream actually publishes: if they say the bug I care about is fixed in version 3.1.4, I can upgrade to version 3.1.4. With distro packages, who knows? The distro might have backported the bugfix into 2.7.1 and bumped the patch number, for all I know. The more places I have to look to find out what's up-to-date and what versions I should use, the more opportunities I have to mess up. Reduce the number of places to look until it's one and only one: the upstream release notes, using upstream's versioning and upstream's packages.

Distro packages for the base OS, language packages for application and its dependencies. Deviate from this at your peril.

[concede_pluto]

If we ship code that runs on SuSE 234 and RHEL 345, that's what we test on. If we run on our own hardware or AWS, we pick a distro and test on that. I don't care whether it builds or runs on a laptop; I don't write twitch games and interesting problems don't fit on a single machine anymore.

It's been a decade since I worked at such a tiny nascent company that all the software was written in just one language. Language packages almost never express dependencies on either system packages or other languages' packages, making "bootstrap to the point of being able to use all of the languages' toolchains" a manual process that lacks any guardrails. Nothing ensures you have a httpd version that's compatible with all your apps, because each of them just sort of assume httpd is out there somewhere without saying anything about it. Staying on the right versions of shared libraries is even more error-prone since the system package manager literally doesn't know you're using them.

If you want to read upstream security advisories and use such bleeding-edge software that even the bleeding-edge distros don't trust it yet, you're basically rolling your own distro that only exists on one machine in the world (because some languages' package managers aren't idempotent and symmetric) and is supported by nobody besides you. I'd rather delegate that to the people who specialize, because the best case is that I don't fuck it up too badly, I'll never add value that way.

[ubernostrum]

If we ship code that runs on SuSE 234 and RHEL 345, that's what we test on.

By all means run the test server as an environment identical to production. I've never said you shouldn't. But people do have to locally run the code on their laptops to do dev work.

I don't care whether it builds or runs on a laptop

Good for you! Now, clean out your desk, because "build custom infrastructure to suit my workflow, but your workflow isn't important" is a clear admission that you don't ever get to work on my team, or probably anyone else's.

I don't write twitch games and interesting problems don't fit on a single machine anymore.

Ah, so you only will work on "interesting" problems, and literally all possible problems you don't find "interesting" are in categories like "twitch games", to be insulted and belittled. It's a good thing you were already fired a paragraph ago, because you'd get fired for that too; turns out most companies don't have problems you'd consider "interesting". So sorry for that, but them's the breaks.

It's been a decade since I worked at such a tiny nascent company that all the software was written in just one language.

With a head that big, what's your size in hats?

Language packages almost never express dependencies on either system packages or other languages' packages

But to actually respond: how many single applications do you think the average company has which are written in, say, five or more different languages and must deploy the entire codebase to a single machine? You know, that single machine you refuse to work with, because it must be just for a "twitch game" or some other tiny puny baby child's toy of a program.

If you want to read upstream security advisories and use such bleeding-edge software that even the bleeding-edge distros don't trust it yet

So sorry that I wanted to use the version with the feature that didn't make it in under the distro's freeze date. Guess we'll just wait ten years until the support contract expires and we're finally forced into an OS upgrade, then? Management will be very happy to hear that timeline, I'll bet they give you a promotion and a raise when they find out you're the one holding it up!

you're basically rolling your own distro that only exists on one machine in the world

I have reproducible builds using language packaging toolchains. Turns out it's 2017 and we can do that now.

I'd rather delegate that to the people who specialize, because the best case is that I don't fuck it up too badly, I'll never add value that way.

There are parts of this sentence that I agree with.

[concede_pluto]

I don't mean to disparage twitch games, that's my go-to example of one of the last domains where it's cost-effective to get really good at living within customers' hardware constraints. But when prod is a growing distributed system, it's natural for tests to assume the same distributed system, and forcing those tests to sort-of run on a single box with the wrong kernel/fs/network config just trades in cheap hardware for expensive engineers doing work that doesn't make prod better.

When I write java, I can't run maven in prod and expect it to get native libraries into /usr/lib64 and the sysadmins' Python and Go plumbing and config files into ... wherever the hell that may live. So together we tweak a .spec file that not only provisions the entire machine correctly but answers questions about whether the entire machine is provisioned correctly, not just the java half. (We probably could make maven do all that, but the result would be worse in every conceivable way, and in most languages it's not even an option.)

If you want a rolling release distro, use one. Godspeed. But nobody's going to sell a support contract that covers random alpha builds published overnight. "We can't upgrade the distro and get the code we need, so we're smuggling in code that the distro doesn't trust yet" is just devs and sysadmins playing chicken over the fate of the project. The bleeding-edge vs supported argument should have been settled before going live.