[adamgordonbell]

If you use chroot to run something, it's interesting how the dynamic libs you need to get in place grows until you are mirroring a whole linux in a subtree. It gives you a sense for how you end up with containers.

One thing that is wild to me is how nix solves this problem, of things needing to be linked together. It doesn't solve it with containers, but by rewriting the location of the links in the executable to be in the nix store. You can run LDD and see it in action.

To me, all that points at containers being in some way a solution to Dynamic linking. And maybe an over the top solution.

Should we be doing more static linking? Not even depending on libc? What are the challenges with that?

[nurple]

Containers are a solution to dependency management for sure. In the world of the FHS, the dynamic linker is meant to solve a number of problems, like space saving and security updates, at the OS level by discovering dynamic deps installed in special library paths. One thing I've never liked about FHS is how everything is organized by kind.

The interesting thing about how Nix approaches the problem is to replace the concept of FHS almost entirely (only a couple binaries are linked to /) by hijacking PATH, and the linker configs like you mentioned. The biggest difference being that the whole version-pinned dep tree is encoded in a nix package(and in the linker config of the binaries it produces) rather than just the package itself.

At some level you could say there is no "dynamic" runtime linking in nix, i.e the linker uses partially specified deps in a discovery phase, all of the link bindings happen at build time.

The FHS did attempt to solve the issue of multi-version dependencies with an interesting name and symlink setup, but they are usually still bound by fairly loose version constraints (like major version). Containers are a lot more like nix in this way, where deps are "resolved" at build time by the distro's package manager by virtue of controlling the process' filesystem.

This is one major issue with the reproducibility of container builds, the distro package managers are not deterministic, you could run a build back-to-back and get different deps depending on your timing(yes, even between test and build CI steps).

[bayindirh]

Disclaimer: I'm not a strong containerization proponent.

The good part of containers is you isolating the thing you're running. I'm very against resource waste, but if I can spend 90MB on a container image instead of installing a complete software stack to run a task which is executed weekly and runs for 10 minutes, I'd prefer that. Plus, I can create a virtual network and storage stack around the container(s) if I need to.

Case in point: I use imap-backup to backup my e-mail accounts, but it's a ruby application and I need to install the whole stack of things, plus the gems. Instead I containerize it, and keep my system clean.

Nix is something different and doesn't solve "many foreign processes not seeing each-other on the same OS" problem. Heck, even Docker doesn't solve all problems, so we have "user level containers" which do not require root access and designed to be run in multitenant systems.

[adamgordonbell]

> many foreign processes not seeing each-other on the same OS

For sure, I was thinking of the packaging nature of containers, not the 'security' nature of containers. The pivotroot part. Though I guess being able to have namespaces does make packaging clearer in certain cases.

For a horrible analogy: With actual shipping containers, we don't have each shipping container be a stripped down model of a ship, so that the things in it aren't confused.

[bayindirh]

The packaging is generally a side-effect of isolation in my experience. I never chrooted a software because I needed a different library stack for it to run, but to isolate it from rest of the system in one way or another (for security/access reasons).

Docker just made the interface more practical, and built the ecosystem around it. lxcontainers, apptainer and podman has improved upon the idea in different ways to cater different use cases.

So for me containers were never simplified ships to begin with. This different perspectives happen probably because people look from different perspectives at initial contact, which is normal.

[samatman]

Agreed, it isn't great. Shipping containers are more like tarballs: a bunch of stuff (files) bundled into one "stuff" (file) so they're easy to ship around and pieces don't get lost in the process.

I don't think there is a good physical-world analogy for what containers are doing, though. Maybe takeout including utensils and napkins to "simulate a kitchen" in case you eat it on the go, so you don't have to rely on the "system" forks and napkins? Still kinda rubbish honestly.

[zer00eyz]

>>> To me, all that points at containers being in some way a solution to Dynamic linking. And maybe an over the top solution. ... Should we be doing more static linking? Not even depending on libc? What are the challenges with that?

>> but it's a ruby application and I need to install the whole stack of things, plus the gems. Instead I containerize it, and keep my system clean.

There was a bit of drama recently with the Bottles project and some Distro maintainers. With the Bottles devs saying "we only want the appimage as a distribution method".

I see containers as a means of distributing software. If that ruby app was a binary, then the container would become baggage.

[bayindirh]

> Bottles devs saying "we only want the appimage as a distribution method".

Good for them having wishes, yet free software doesn't work that way in most cases. :)

> I see containers as a means of distributing software

That's OK. Docker is a tool, and tools can be held and used in many ways.

[zer00eyz]

>>> Good for them having wishes, yet free software doesn't work that way in most cases. :)

Historically I would agree with you.

But bottles, and the attitude of some upstream projects is one of "modern", "faster", "support the latest version only" ... Software devs who live in user space want a more "App Store" like delivery mechanism where they have control. The kernel is inclined to never break backwards compatibility. Distros are sort of stuck in the middle...

>> I see containers ... > That's OK. Docker is a tool,

One of these is a subset of the other... (depending on your perspective). Dockerdesktop running a linux image on a windows box is most certainly a tool... that same container being ported to lcx in production makes it a package manager. Who is a subset of whom is something we could debate but these things can be both.

[prmoustache]

Honestly I don't understand that obsession of developers with building and distributing binaries themselves. That is the less fun part.

When I write something, I just release the source code and I let other do the non fun part for me. You just have to mention the dependencies and how to build and that's it.

[EraYaN]

Because it's easier to support, easier to answer bug reports and/or crash reports etc. And distros can now not mess up the build too much, which happens a lot with any sufficiently complicated piece of software. Besides your average user does not want to deal with building something like OBS or Ardour or any of these programs that have a ton of dependencies and no real unified package manager to install them.

[bayindirh]

> Software devs who live in user space want a more "App Store" like delivery mechanism where they have control.

They can have control. They can say that we have flatpaks and app images we publish and support, and they only support the latest version (or the same minor version, whatever), and can say that packages got from distros may not be the latest.

There's no need for a yelling match, IMHO.

Distros also can do whatever they want. Like rclone. You can get the packages from rclone.org or from your distros. There's no yelling match, but trade-offs.

> ...that same container being ported to lcx in production makes it a package manager.

I don't think so. I have containers which work like binaries (in the form of "./binary infile outfile" fashion) and exit after processing what I give them. For me that container is a utility program as a whole. Same for the imap-backup example.

When you think services which are always on, docker might be a package manager, but I pack my own containers for example, so it's more like compiling for me.

So, what docker or containers is depends on your perspective, or like a chameleon which changes its color according to the landscape it's in.

So, it's a tool in the end. Package managers are tools, too.

[sloowm]

I think it was Flatpak that Bottles prefers.

The thing with Bottles is that it also relies on a very specific setup of all underlying software. So in that case it's more about complexity that leads to the need of containerization than keeping systems clean or containerizing for sandboxing.

[marcosdumay]

Linux doesn't solve that problem at all. Qubes tries to do that with a hack, that is probably sufficient, but quite complex.

I'd say that we don't have good solutions to that problem. But nobody even tried our best candidates, so I'm not completely sure. (For a while, it looked like Android would finally try some. But then Google turned it into a user-hostile anti-privacy OS.)

[tutfbhuf]

One issue with static linking is that your dependencies will likely have critical CVEs over time. If you keep all your libraries separate on the filesystem, you can just do a "apt update; apt upgrade", and you will have all the latest patches. This will patch security issues in e.g. libssl or libc for all your applications that are dynamically linked against this shared libraries, which can be quite a few. In static binaries, the version of the libraries is not obvious from the outside. If you have, for example, 100 fully static binaries, these can come in 100 different major/minor/patch level versions of their dependencies. You now have to patch each binary separately by upgrading and recompilation 100 times to patch all your static binaries, that requires much more time and energy.

[adamgordonbell]

That all makes sense. But when those 100 binaries end up as 100 OCI images, and then to patch them you need to update those 100 OCI images to have the new version, it does seem like we've gone in a circle a bit.

I mean, there are some advantages, if they all share the same base layer, maybe they share those libs at least on disk via a shared layer. But practically, though you are maybe not back where you started, you are at a place that seems to share some similarities.

[nurple]

This^

This is one of the biggest issues with containers IMO. This and the layering system, which I think is poorly designed both to configure and to actually do the tasks it's meant to(build and delivery caching).

The solutions to this problem in the space have basically been to provide scanners to crack open the containers and detect things with known vulnerabilities. But I have not seen (m)any solutions around these scanners to facilitate the lifecycle of landing fixes.

Even if you provided a tree of a-proved base containers for each deployment lang in your org, you can't just update the base and deploy the world, there's not even tooling to automate working over the "FROM hierarchy" of images where you could detect which need to use new bases.

Because of the difficulty in managing large container hierarchies, in some orgs the later drives a common methodology of making image tags mutable, i.e. `ruby:myorg-v2`, which makes the FROM more like a dynamic link reference that gets updated automatically on the next build. I view this workaround as a regression brought on by the _still_ incredibly poor and complex SDLC tooling around managing images.

[rdtsc]

It's like for a few years the whole computing world forgot about security updates and went on a container bender, when woke up one morning realizing what they've done, and then started adding various clunky solutions on top: mutable tags, various tooling to take apart container images and inspect the junk in there, notify and scan for issues etc.

[pjmlp]

You mean like proposing JSON as XML replacement, rebuilding from scratch all the XML tooling including validation, and when almost done with it, replacing JSON with YAML?

[rdtsc]

Exactly! YAML apparently is now terrible as well, but not sure what's next in the pipeline, toml perhaps?

I think it would be funny if we reverted back to s-expressions and started the cycle all over.

[tutfbhuf]

You are right. If you put 100 dynamically linked binaries into 100 OCI images, then you have the same security issues all over again. As best practice, I would recommend using a container vulnerability scanner that can identify containers requiring updates (list CVEs). I think all major cloud providers have such a service available, and there are some free and open-source tools available, such as Trivy and Clair. It is also beneficial to use official container images that have frequent patches available for their base images. If you use a base image like 3.9-slim instead of 3.9.19-slim, you can, for example, pin your Python version to 3.9, but you get patches. But this again only works if you do not have a "FROM scratch" image with just a single fully static binary.

[jayd16]

Containers are powerful because they solve many computing issues, one of which being able to act as a (lower case c) static container for dynamically linked apps as well as cross-language, multi-executable meta-apps.

Containers also provide many forms of isolation (network, file system etc.), they provide a modern versioning and distribution scheme, composibility (use another container as a base image).

All of these things can, and perhaps should be, done at a the language level as well but containers also work across languages, across linking paradigms, and with existing binaries.

[nonameiguess]

I haven't seen any other response mention it yet, but containers are also heavily used for web-exposed services in part because of address space and port contention. Network namespaces allow you to graft an overlay network onto your physical network in a relatively simple and easy way (not that it's actually easy, but networking never is).

Otherwise, sure, nix can rewrite the RPATH in your ELF file to make it pull dynamic libs from the nix store, but what does it do when two processes both want to listen on ports 80 and 443?

Possibly, if the Internet ever actually goes pure IPv6, one LAN will have enough addresses to assign one to each process instead of each host.

There are, of course, other ways to handle it. People used vhosts predominantly defined in a dedicated web server that was really only a reverse proxy, but now you need nix and nginx. Then you discover you also want resource isolation. Is there a userspace alternative to cGroups? I don't see how there could even in principle be an alternative to PID/UID namespaces and UID/GID submapping. Some things have to happen in the kernel and that means containers of some sort. It doesn't have to be the exact OCI standard that eventually grew out of Docker and eventually Kubernetes, but some kind of container.

[adamgordonbell]

I think what you are saying is true, and my knowledge of networking is pretty slim.

But, to play along with my static linking thought-exercise: if you take a process and put it in a network namespace then is it a container? I wouldn't say it is. Container runtimes might have a nice interface for namespacing, but namespacing something doesn't make it a container.

I guess my thought experiment is if things are statically linked binaries and you had a way to run them with the control group and namespace settings you wanted, would the packaging aspect of containers add anything?

The elites don't want you to know it, but namespaces are just there for the taking. You can grab as many as you want. You can set the memory limit on any process with cgroups, no docker desktop required. :)

Anyways, just a thought experiment about how the industry sometimes seems to be going in a circle, in the fashion of the lady who swallowed a fly.

[Already__Taken]

dynamic linking to me seems like solving a problem that filesystems should. deduplicating data.

[sltkr]

They are complementary. If you want to reuse the shared parts of binaries, then you need a way to separate the binary image into parts that are specific to the application and libraries that can be reused across binary images, plus some metadata how to reconstruct the image from its parts. That's what exactly what dynamic libraries provide.

In general, it's much easier to link a binary with its libraries than go in to the opposite direction (extracting common code from static linked binaries), because once you statically link a binary the library code will vary slightly due to differences in memory addresses, compiler optimizations, unused code that has been omitted, different input library versions, etc.

Even if you were, in theory, able to write a complex filesystem driver that is able to extract the common parts of statically linked libraries so they can be deduplicated, then to reconstruct the original binary in memory, you'd have to perform something similar to dynamic linking, except now in the kernel, which really isn't an improvement.

[adamgordonbell]

> If you want to reuse the shared parts of binaries

But aren't we, when we use OCI images as a packaging mechanism, using containers to essentially throw away that sharing and arrive at a complicated version of static linking, where everything dynamically linked is shipped with the program?

Same goes for arguments about ease of patching things. When the software's package is actually an image, you are patching each image individually that is running on the system.

[sltkr]

No, because you can share common libraries across containers by putting them in a separate layer.

For example, if you have a complex service that consists of multiple binaries all written in C++ using boost, then for each binary you can create a container that contains a layer of a base OS (shared), C++ libraries (shared), boost libraries (shared), application binary (unique).

All the services can now share their common libraries, both on disk and in memory, which reduces I/O and memory use. That's one of the main advantages of containers over virtual machines (VMs): each VM instance has a distinct region of memory that is not shared with others even if they happen to load bit-for-bit identical binaries into memory.

(I know, VM memory deduplication exists to ameliorate this problem, but here my previous comment applies: it's much easier to start from shared components and link them together than extract the shared data after the fact. And typically VMs have lots of nonsharable state that containers do share, like pretty much all writable kernel pages.)

[ahepp]

Are you saying that two containers running the same image will share their common libraries in the host kernel's memory?

Based on my understanding of cgroups, that seems unintuitive to me. Are you certain that's the case? I may try testing this out when I get a chance.

[sltkr]

Yes. And even containers running different images will share the libraries so long as they come from shared layers.

[morcus]

> All the services can now share their common libraries, both on disk and in memory, which reduces I/O and memory use.

Wow, how is this possible using layers? How does docker handle it if I subsequently modify one of the files of my layer in my container?

[sltkr]

A docker image consists of several layers, each of which contains only the modifications to the layers below it. Each layer and the final image is immutable. Docker uses OverlayFS to provide a unified view of the various layers.

A running container is based on an immutable image and a single writable layer. That writable layer is unique to the container which contains all modifications made to the immutable image by processes running in the container.

Docker relies on the immutability of layers to share them between containers. This is not much different from how regular Linux processes all share the readonly contents of binaries and libraries that they load, while each process has its own private heap space that is not shared with other processes.

That means that deleting a file from a base layer, either when building an image or at runtime from the container, doesn't actually modify the contents of that layer. It only adds a tombstone marker to the writable layer, that indicates the file was deleted, and OverlayFS creates the illusion that the file no longer exists inside that container.

(The flipside is that deleting files from immutable layers doesn't actually free up space because the actual file contents don't go anywhere, but that's rarely a problem.)

[forgotmyinfo]

Someone did once joke that Docker is just static linking for millennials. I chuckle because there's a kernel of truth in there. ;)

[wredcoll]

Yeah, maybe, but the cute thing here is that docker also "statically links" the config files, history db, image directory, etc etc.

[forgotmyinfo]

In the same way a zip file is static linking, I guess, since Docker images are just tarred gzips. But .deb packages can include files and scripts that create directories, too.

[pjmlp]

Available in static linking via resource files.

[bayindirh]

I think a better analogy is, with containers you maintain more "servers" in the end, and patch and reboot them all.

So there's no free lunch and everything is a trade off. People thinking that containers are no more work than managing servers or services are in the wrong.

[Rucadi]

OCI is not the problem, you can create reproducible images using NIX that reuse all the packages between different images.

The problem is the way we usually package the apps with Docker-Dockerlike builders

[palata]

They do more than that. For instance, they can be swapped easily. Think for instance a security library being updated by the distro security team. It also makes it easier to depend on an LGPL library, nicely allowing the users to modify the LGPL library without having to recompile your program.

[zer00eyz]

>>> Think for instance a security library being updated by the distro security team.

And when your distro doesn't update? When the distro is slow to change?

Flatpak, Appimage, Snaps (why?)... upstream devs are bypassing distro maintainers in a lot of cases.

[palata]

I am not saying that they solve all problems in the world. I am just saying that they have merits.

In your particular example, if you are not happy with your distro, you are free to find another one. Some distros are extremely fast to change.

> upstream devs are bypassing distro maintainers in a lot of cases

IMO, for open source projects, upstream devs should not distribute their library. They should let package maintainers do it for their distro.

[forgotmyinfo]

Then there's probably a reason, like the changes haven't been tested yet or verified to make sure they don't FUBAR your machine. It's okay to go a little slower to ensure reliability. Plus, do you really want the latest bugs in HEAD anyway?

[bayindirh]

As a bonus, shared libraries are deduplicated in RAM, too.

Plus you don't need to recompile the universe because of a patch.

[otabdeveloper4]

No, mostly it solves the problem of runtime library loading and swapping. E.g., Linux VDSO can only work as a shared library.

[brabel]

That's an interesting way to look at it. If all dynamic libs lived on a read-only location, then the file system could actually only store the libs in one place and the other "copies" would be just symlinks to that... and when the OS loaded such lib, it would automatically know that despite being in different locations, the libs were the same (they're all symlinks to the same place). Is this something that has been attempted before?

[surajrmal]

You can store the files as if you name the actual location of the file as a hash of its contents and symlink the file to that location, you naturally get deduplication. Fuchsia does this [1]. You still end up wanting to try and coordinate your packages to share as many deps as possible for resource optimization reasons, but you no longer depend on it.

[1]: https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs

[chrisweekly]

Slight tangent, but in the nodejs / npm ecosystem, that's one of the things that makes pnpm unique (and IMHO far superior to npm or yarn) -- its node_modules are deduped using symlinks.

[nurple]

This is exactly how nix works, except instead of symlinks it actually modifies the binaries at buld-time to point its dynamic library paths to absolute paths in the store which includes a source-derived hash in place of a version.

[forgotmyinfo]

This is basically dynamic linking: you have foo.so sitting somewhere on disk, and multiple processes can just load it whenever. It can't be read-only, though; you need to add new libraries to that directory eventually.

[pjc50]

Traditionally this is what /usr/lib is for.

[fl7305]

> Is this something that has been attempted before?

Yes, some file systems implement "deduplication".

[ahepp]

Deduplication, in the sense of physical space reduction, seems like the least strong argument one could make for dynamic linking these days.

The strongest argument I can think of, is enabling system managers (distro maintainers, even end users) to update dependencies. This might be to apply a security patch, enable some kind of tracing for profiling an application, and so on.

[watermelon0]

In practice, dynamic linking might not be achiving that much in the sense of deduplication: https://drewdevault.com/dynlib.html

[forgotmyinfo]

Filesystems solve linking insofar as the libraries themselves are on disk; otherwise, this is squarely the responsibility of the loader.

[pjmlp]

Plugins have nothing to do with filesystems.

[silverquiet]

Funny - it looks like you’re being downvoted for asking what I think is a very natural question. It’s one I’ve asked before; have we just created a more elaborate statically-linked executable via containerization? In the end, Docker/OCI seems like the universal Linux package manager.

I’m sure I don’t have the full picture since I’m far more ops than dev though.

[bayindirh]

I don't think so. Instead we created an artifact which can live inside an OS, but cannot see and touch to the rest of the OS.

CGroups is a deceptively powerful mechanism. You can isolate a process resource wise (X cores, Y amount of memory, Z amount of swap), network wise (a different virtual network adapter with its own IP, bandwidth limits, etc.) and FS wise (running in its own filesystem with devices it can see).

It can keep your system tidy by encapsulating elaborate stacks which makes system management painful, allows deterministic operation and image generation if you tag everything with version.

Downside is you can do bad things with it like terminating HTTPS with a gateway container and talk HTTP among your backend instead of configuring tools, or writing shoddy software, and getting away with it because it works, or gets automatically restarted when it crashes 6 times a day.

I don't run every service as a container, because some services suffocate when they are in a container, but for short running things which needs system-wide changes to function, or test-driving small services before fully committing, it's a good tool to have.

However, it's abused with no end, and their popular use leaves a bad taste in your mouth.

[adamgordonbell]

You can put any given process is a cgroup. It doesn't have to be a container.

If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well.

No `FROM X` `RUN Y` dockerfile stuff needed.

[bayindirh]

> You can put any given process is a cgroup. It doesn't have to be a container.

Yes, I run many programs inside cgroups, but not in containers.

> If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well.

Yes.

> No `FROM X` `RUN Y` dockerfile stuff needed.

Yes. dockerfile only sets up the chroot in an overlayfs and fires up the "container" using mechanisms present in the kernel already.

As I said on another comment, quoting myself:

> Docker just made the interface more practical, and built the ecosystem around it.

[zer00eyz]

If you have a go/rust/zig binary as your application do you need a container to run it?

Maybe, but it makes less sense at that point. If your doing Node/Ruby/Python/PHP then yes the container makes sense to drag your runtime to the server...

Do containers (docker) make sense for dev. Sure, to a point. Because our dev (win/Mac) might not look like our deploy (linux).... If we move to a standardized remote dev model then docker makes less and less sense.

>> It’s one I’ve asked before; have we just created a more elaborate statically-linked executable via containerization?

The bottles project only supports their app image, as they no longer want to be responsible for supporting the disttro maintained packaged version of their product.

Yes containers are becoming a way of dealing with linking, and dependency management. Its a blunt instrument for dealing with software packaging and distribution.

[ianburrell]

The huge advantage of containers is that can use the same mechanism to run anything. Don't care if it is a single Go binary, dynamically linked binary, or Ruby interpreter and thousand files.

You might start off writing Go programs, but then need to run Postgres database for development. Or discover that need special library in some other language and easier to make its own service. Or need to run third-party service. With Docker, you run the image and don't care what's inside, and the isolation gives some assurance that won't escape.

[magicalhippo]

For me as a mere user, wanting to run some homelab services, the main advantages to containers are that they make updates easier (don't need to wait for distro), and it makes it much clearer where configuration and data lives, easing backup and rollback by orders of magnitude.

Static vs dynamic linking is an implementation detail as far as I'm concerned. If all the dynamic libs needed were in a well-defined location it wouldn't matter that much.

[forgotmyinfo]

The benefit of waiting for maintainers to update your software is that you have a stronger guarantee that it won't break your system, or otherwise fubar something. Maintainers are the adults in the room saying "no, fix your shit" when sloppy developers release crap, which seems to be happening more and more frequently lately.

As for where configuration and data live, that's always available in the docs, and Linux convention puts stuff in /etc, so I'm not sure how containers help. And dynamic libraries are in a well-defined location, with environment variables and other tools that allow you to specify where they live. It's not like dynamic linking is an unsolved problem.

[EraYaN]

Maintainers are also the ones breaking software so, realistically the difference is basically moot. And for a containers to fubar the system you have to really mess up. At worst that specific container fubars itself and you rollback a tag.

There are just fewer things that can go wrong when you get to a sufficient number of services. And lastly moving to a new host is infinitely easier too, export the volume, import of new host and off you go. And stuff like Kubernetes will just handle this for you (and more).

And as for those linux conventions, they vary a lot from distro to distro, you can never be quite sure where that specific version of that specific distro puts its files. So having them just not be able to touch the host ever is a good thing.

[c-hendricks]

Right, but a piece of software having a distribution maintainer doesn't mean it will never have bugs, and if it's a container there's already much less risk of it breaking my system.

As for your second paragraph, that's very idealistic. Config can live in /var, /usr, /home, /usr/local, literally anywhere. I find it much nicer when all data / configuration for a piece of software is all self contained.

[silverquiet]

I think of myself as a mere user as well, though I manage the container system/orchestration for a small SaaS company as well (we're weirdos who use Swarm instead of Kubernetes) and agree with you regarding the management benefits.

[pjmlp]

That wasn't the deal in HP-UX vaults, Solaris Zones, System 360/MVS virtualization,...

The way they are used on GNU/Linux is indeed a solution to GNU/Linux's software distribution issues on a highly fragmented landscape.