[Annatar]

At the very least (npm, conda) multiple independent environments, allowing conflicting versions of a library to exist simultaneously in different environments.

One should never, ever bypass the operating system's software management subsystem, because then, one needs a hack like Docker, since the OS will have no notion of the software installed, and a system specifically invented and designed for managing said software won't be able to manage it. I could author a book on just how bad of an idea using a packaging system proprietary to some language or application is. By utilizing a packaging system outside of the operating system's packaging, anyone doing so undermines operational repeatability (see capability maturity model level 2 and higher).

allowing conflicting versions of a library to exist simultaneously in different environments.

That's another very bad thing: it's a workaround for lack of thinking about backwards compatibility, and designing a single library with multiple versions of the API, and, and, and... I happen to work on an application which utilizes 300 such environments, and it's a nightmare.

Finally, by using a packaging system proprietary to a language, anyone doing so is now forcing everyone else who wants to use said software to wrestle with multiple packaging systems; and if every developer thought that way, the end consumer would have to wrestle with all of them. I don't think I need to point out just how irresponsible, not to mention inconsiderate that is towards users. Users of the application, not developers should come first, and only then everything else. In this day and age of automated builds, it is trivial to set up automatic generation of operating system packages for various systems. Trivial, and developers must take responsibility for that, instead of putting the onus on the consumer of the software.

[beagle3]

> One should never, ever bypass the operating system's software management subsystem,

Neither conda nor npm does something like that. They set up a localized environment, insulated from the host os installed software as far as packages they manage are concerned.

> By utilizing a packaging system outside of the operating system's packaging, anyone doing so undermines operational repeatability (see capability maturity model level 2 and higher).

No they don't. I'm not sure about NPM, but conda documents exactly what packages are installed inside it, generates a specification file you can use on another install to recreate the same environment, and insulates the inside install from changes in the underlying OS.

> That's another very bad thing: it's a workaround for lack of thinking about backwards compatibility, and designing a single library with multiple versions of the API, and, and, and... I happen to work on an application which utilizes 300 such environments, and it's a nightmare.

Feel sorry for you. But I have to maintain two projects, one requires an older version of the library, one requires the newer. There is no way to achieve this with "apt-get" or "yum". Conda lets me develop both on the same machine in a repeatable way. And it's way better than any other solution I'm aware of.

> I need to point out just how irresponsible, not to mention inconsiderate that is towards users. Users of the application, not developers should come first, and only then everything else. In this day and age of automated builds, it is trivial to set up automatic generation of operating system packages for various systems. Trivial, and developers must take responsibility for that, instead of putting the onus on the consumer of the software.

What?

There are many packaging systems for Linux/BSD/Unix I use often (apt/deb, yum/dnf/rpm, pacman, ports, brew, cygwin), even more repositories (ubuntu, debian, ...) and there is absolutely zero chance that I can rely on dependencies to be provided by the OS package manager - they don't update at the same rate, and sometimes not in a compatible way (e.g. PIL vs. Pillow migration for Python; ffmpeg vs libav).

Conda, however, makes that a simple "conda install" away, once you have conda installed. And it also works on Windows - which has no canonical package manager (chocolatey is the closest, but it is still very far).

What you are suggestion is useless in my situation, and for at least 95% of the developers and deployment situations I'm familiar with. If it works for you, then ... good for you. But you're pretty unique.

[Annatar]

Neither conda nor npm does something like that. They set up a localized environment, insulated from the host os installed software as far as packages they manage are concerned.

That is exactly what I mean by bypassing the operating system's software management subsystem. No software or configuration should ever be installed by anything other than the operating system's software management subsystem. It is the responsibility of the developer (you or anyone else doing so) to convert content pulled in by npm or anything that is not an OS package into one, but the best solution is to fix it upstream by abolishing these parallel software management subsystems, and automatically generating native OS packages for the software through continuous integration build processes.

Not doing so puts the onus on the consumer of the software, and to wit, the purpose of software is to make it easier, not harder or more time consuming, to do something. Imagine forcing the system administrator to have to replicate your development environment, through any mechanism, just to be able to run your application and all her dependencies, when she or he could have just plugged in OS packages of your software into the OS provisioning repository.

Feel sorry for you. But I have to maintain two projects, one requires an older version of the library, one requires the newer. There is no way to achieve this with "apt-get" or "yum".

Actually, yes there is: make the library interfaces versioned, and fuse them into one library for all versions of the consuming software. This is, post action, considerably more work, but that is what developing software is all about. The idea of versioned interfaces is not new, and originates on UNIX. Since the initial topic of discussion is building and linking libraries in C, the link editor can consume linker map files, which define the library's semantics based on the version of the interface, and will encode this information into the executable and linking format header of the binary. The encoded information will be consumed by the run time linker in order to present the correct version of the interface to the consuming application. Same principle applies to any software in any programming language.

There are many packaging systems for Linux/BSD/Unix I use often (apt/deb, yum/dnf/rpm, pacman, ports, brew, cygwin), even more repositories (ubuntu, debian, ...) and there is absolutely zero chance that I can rely on dependencies to be provided by the OS package manager - they don't update at the same rate, and sometimes not in a compatible way (e.g. PIL vs. Pillow migration for Python; ffmpeg vs libav).

Then the developer must provide the additional dependencies as OS packages, and can do so without interfering with the OS vendor by adhering to the System V / Linux Standards Base - Filesystem Hierarchy Standard: by delivering the payload in /opt[/namespace], the configuration in /etc/opt[/namespace], and coding the application to use data from /var/opt[/namespace]. The fathers of UNIX considered this, and provided specifications to address it accordingly.

http://refspecs.linuxfoundation.org/fhs.shtml

What you are suggestion is useless in my situation, and for at least 95% of the developers and deployment situations I'm familiar with.

As you can probably imagine, I disagree vehemently with your assertion, and it is my hope that through this dialog, I've managed to illustrate the reasons because of which that is so, as well as provided practical solutions to the problem at hand.

If it works for you, then ... good for you. But you're pretty unique.

You mean, I am unique in wishing to deliver software to the highest standards of quality enabled by our current technology? Having started as a system administrator, being a developer, I can completely understand the gargantuan amounts of work system administrators face trying to make developers' work repeatable, essentially being forced to complete developers' work for them. And that is simply not just, nor is it right. We as developers should do that.

Empathy is a core engineering value:

https://www.listbox.com/member/archive/182179/2013/05/sort/t...

respect your users and empathize with them. If you don't quite know how to solve the problem, ask system administrators to help you produce higher quality software, or even better, learn system administration yourself, just don't punt on the problem because you perceive it as too much work.

[beagle3]

> Actually, yes there is: make the library interfaces versioned, and fuse them into one library for all versions of the consuming software. This is, post action, considerably more work, but that is what developing software is all about.

You assume I am the library maintainer, but I am not. I am just the library user. ergo, no amount of apt-get will fix it for me.

> Then the developer must provide the additional dependencies as OS packages, and can do so without interfering with the OS vendor by adhering to the System V / Linux Standards Base - Filesystem Hierarchy Standard: by delivering the payload in /opt[/namespace], the configuration in /etc/opt[/namespace], and coding the application to use data from /var/opt[/namespace]. The fathers of UNIX considered this, and provided specifications to address it accordingly.

No amount of apt-get will do that if the library/distribution/upstream developer didn't.

> You mean, I am unique in wishing to deliver software to the highest standards of quality enabled by our current technology? Having started as a system administrator, being a developer, I can completely understand the gargantuan amounts of work system administrators face trying to make developers' work repeatable, essentially being forced to complete developers' work for them. And that is simply not just, nor is it right. We as developers should do that.

Your solutions only work if you are the packager for every piece of software you use. That is a unique situation. Most of us have to use packages written and maintained by others.

Furthermore, Docker makes everything work magically for system administrators, and yet you are against it. Why?

And conda makes everything much more repeatable than the underlying OS apt-get can; I have the same, perfectly repeatable, conda-base app running on ubuntu 10 (yes, it needs to go), ubuntu 12, ubuntu 14, ubuntu 16, debian 6, debian 7 and debian 8 right now. It is impossible to do that with underlying OS package management - but conda is perfectly repeatable.

I've single handedly administered north of 200 servers of various makes (linux, windows, solaris), I understand the need for repeatability, and I get it from conda. I can't get it from the underlying package managers. My experience is so different from what you are proposing that I suspect we are coming from different worlds.

[Annatar]

You assume I am the library maintainer, but I am not. I am just the library user. ergo, no amount of apt-get will fix it for me.

Then take the Apple Computer, vertically integrated stack approach: package all the libraries you require into OS packages, so that the entire installation is encapsulated.

No amount of apt-get will do that if the library/distribution/upstream developer didn't.

How about working with the upstream developers on it? But I agree on this point with you: you, as the user, have a right to expect to be treated with dignity and respect: if the developers aren't providing OS packages for you, and they are on the same platform as you, and refuse to do so in favor of making their lives easier, then that's just wrong.

Furthermore, Docker makes everything work magically for system administrators, and yet you are against it. Why?

Because it doesn't. Docker makes a system administrator's job a living hell: when production kicks the bucket, and the system administrator is called in the middle of the night to troubleshoot the application, he cannot ask one subsystem designed for this purpose to tell it about the software: the software management subsystem. System administrators hate being burdened with artificial dependencies which make developers' lives easier, but make system administrator's jobs living hell. Change and configuration management with Docker is hell, because it doesn't exist. Docker isn't designed to support the change management process, because the developers of it have never heard of such a thing. How would you model the change management process with Docker? What about software lifecycle management, roll a several gigabyte image every time you need to fix your database component? What if you have thousands of such applications that you're responsible for? And why should you be forced to use Docker, when there are zones which provide far superior containerization?

As if that weren't bad enough, if developers delivered their software as modular units -- as OS packages, Docker would be completely and utterly unnecessary. In fact, it's unnecessary now, and only artificial insistence from developers makes it necessary. This is not a good situation.

Dumping a bunch of files into a format foreign to the operating system is never a good solution. OS packaging is designed from the ground up and optimized for managing software. Provisioning technologies like JumpStart, Kickstart, AutoYaST, Ignite-UX and NIM make it possible to go from bare metal to fully functioning systems, ready to accept and serve data. When one has OS packages, all one needs to do is plug them into systems already optimized for that. No need for arbitrary 3rd party solutions.

Packaging was and is, actually, in the strictest sense, designed for software developers, and yet, I have worked with many, many software developers who actively fight even looking at how to make an OS package, for no better reason than learning something new, and yet they'll spring for every shiny new programming language which shows up.

I've single handedly administered north of 200 servers of various makes (linux, windows, solaris), I understand the need for repeatability, and I get it from conda. I can't get it from the underlying package managers. My experience is so different from what you are proposing that I suspect we are coming from different worlds.

I don't know what to tell you. I have almost 73,000 systems across various countries around the globe, and growing daily with no end in sight. I specialized, and have formal education in UNIX system administration (in addition to formal education and professional experience in programming and algorithms). Several decades of professional experience in both, plus database engineering experience, plus large scale system engineering and architecture experience, as you can probably infer from the number of systems. But really, I don't think our worlds are that different, I don't think they're different at all: if it works on almost 73,000 systems, it will work on "north of 200".

[beagle3]

Let me clarify. I managed, on my own, on metal, north of 200 in 2006, as a developer, before all modern orchestration software and even EC2 wasn't even announced until later that year. And when that started taking a nontrivial amount of time, I hired a sysadmin (which is not my job). I didn't bring it up as a pissing contest, just to indicate that I have relevant experience and appreciation of consistency. It's great you have 73,000 in your fleet, but I'm not at all sure you have more work to do than i did with 200 a decade ago.

That said, we do live in completely different worlds. I deliver software that works, repeatedly, on many flavors and historical versions of Linux, some out of vendor support or that never had vendor support, and windows. And does that without inconveniencing the user, and without costing me much in development time as to make it uneconomical.

None of your suggested solutions make any sense in my world. I need to run python 3.5 and a variety of its packages, but the client runs ubuntu 10.4; you suggest I either get the python maintainers to compile it for 10.4, or the ubuntu devs to, which couldn't have happened even when 10.4 was still supported; and then do it for every package I need. Alternatively, you want me to compile and maintain it myself as an 10.4 package, that shouldn't interfere with system python - but also 12.04, 14.04, 16.04 windows and a couple of centos. Which would also make me liable for tracking upstream security for every package I use, rebuild and distribute.

Alternatively, I can ask my customer to install miniconda (one download, one command to execute, regardless of platform), and then install my package through conda (two commands - create env and install). No interference with system packages, no breakage when they upgrade their system to a newer distribution.

Where as in your world you can run a mini Apple to vertically integrate all your packages, take responsibility for security updates for code you don't own, possibly force your customers to a uniform setup (and let go those who don't/can't).

Totally different worlds.

[Annatar]

but I'm not at all sure you have more work to do than i did with 200 a decade ago.

I don't, and that is exactly my point: it's not more or as much work as you perceive it to be, making one's own packages. An experienced packager can get 98% of the software packaged in under five minutes. I do that all the time.

I would have never picked Python precisely because of the pathologies you describe (and which became obvious to me when building and packaging Python). In my experience, your choice of the programming environment led down the rabbit hole you describe.