But, if ld.so supports $ORIGIN already, why does the kernel needs to support it also? Or rather, why can't the kernel leverage ld.so and do this entirely on userspace?
$ORIGIN was only supported when the loader was looking for other dependencies.
The loader itself (field PT_INTERP) is loaded by the kernel. So prior to this change, every program must hardcode the absolute path to ld.so.
With support for an $ORIGIN-relative loader, each program could use its own copy of ld.so.
Each program having its own loader is an anti pattern. Such a requirement is overkill. You can have a single loader that supports everything on the system.
Not with NixOS. ld.so is tied to a version of glibc, in ways that can be subtly incompatible. And nixos can have multiple glibc version installed on a single machine.
Besides, it allows for upgrades/downgrades to be done in a way that's much less error-prone.
$ /lib64/ld-linux-x86-64.so.2
/lib64/ld-linux-x86-64.so.2: missing program name
Try '/lib64/ld-linux-x86-64.so.2 --help' for more information.
!1!
$ /lib64/ld-linux-x86-64.so.2 --version
ld.so (GNU libc) stable release version 2.34.
^^^^^^^^^^^^^^^^
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
There is much more value to be had in making glibc properly backward compatible so you can have a single one that can be used with everything than trying to make it so that you can swap everything around, creating extra complexity and compatibility risks.
> […] in making glibc properly backward compatible […]
You could only do that going forward though, and would be stuck, at least for a time, with the historic versions that still need extra handling. In an ideal world that wouldn't be needed, but in an ideal world you'd not need multiple versions of glibc at all so we aren't there.
Support in the kernel means that it will work even if applied to old packages that for some other reason need a held-back version of glibc. As mentioned elsewhere it also gives the feature to #! directives in scripts too.
How is that more valuable? It comes with two big pitfalls:
- It would still not allow downgrades to work properly
- It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together
Meanwhile, having different ld.so has many good use-cases, like simplifying development of ld.so itself, allowing them to swapped during updates in ways that are safer, etc...
And the eBPF binfmt support is a rather simple, generic mechanism that is likely to have many other use-cases beyond ld.so. So it's overall a pretty good resolution to the issue?
Unfortunately, Glibc and loader are linked together intrinsically in the Linux ecosystem. So, if you want to be able to launch a program reliably, shipping your own loader and libc might actually be the only way.
Glibc is backward compatible though, they even have symbol versioning to provide multiple versions of the same symbol. So as long as you have the same or a newer version of glibc (than what was built against) you should be good to go. And I don't remember hearing about breakages for this.
Other libraries on the system is far more hit and miss, but glibc is quite compatible.
The other way around is harder though, you can't take a program built against a newer glibc and run it on an older version. So you generally need to spin up a container with some LTS distro and build your binary in it if you want it to be maximally compatible. (However, zig apparently is able to deal with this by shipping a mapping between glibc versions and symbol versions and doing the linking themselves. You can even use zig to link rust code using cargo-zigbuild and get that benefit.)
But if you want to be maximally portable: static linking against musl. Though beware that many things are slower in musl, such as the allocator.
No, glibc isn't backwards compatible; I've had instances when the loader would refuse to load the executable because the installed glibc is too new for it.
> why can't the kernel leverage ld.so and do this entirely on userspace?
Because Linux is completely independent from its user space. There is no "the loader in Linux", it just happens that GNU wrote one and it's widely used.
You have the right idea but the wrong specifics. The boundary you're alluding to isn't the kernel/userspace boundary, it's the `libc` boundary, which is admittedly privileged by convention if not by Ring 0.
On most Linux systems this is regrettably `glibc` (in a container where you get to choose everyone chooses the superior option of `musl`), on all Darwin systems this is `libstandard`.
This thread is more concerned with Linux, so you are probably dealing with `/lib64/ld-linux-x86-64.so.2`, which operates in userspace but is by convention and opacity quite clearly part of "the system".
The kernel modification is a much bigger, much weirder side effect of a weird Nix loyalty test around shebang lines in shell scripts, on which is has opted to be intentionally and violently incompatible with everything for no benefit other than incompatibility.
That’s a weird rant which is manifestly false. On Nix the benefit you get from the incompatibility is the system can roll forward and back while it’s running so you can do big changes (eg updating or rolling back) the entire system without affecting running processes. You can also run groups of processes in isolation with different sets (or versions) of packages from the rest of the system.
It’s fair to argue about whether those things are really important or whether nix does this in the best way, but to claim that the only benefit is incompatibility is just obvious nonsense.
The privileged path is already there. `/usr/bin/env` is no different from `/bin/bash`, `/bin/sh`, or any other ELF artifact at a known place. The argument is made, the argument is spurious. It is made in the other direction regarding the driver run path, just as religious, opposite ruling from purity court. No one even knows why, the trail goes cold in a mysterious 2012 commit about Mesa, it's literally a performance to bring the plane cargo back. Receipts for claim: #141803.
Disagreeing with you doesn't make something a rant. If it's not clear how deep my Nix expertise is I will demonstrate it to any level you like: I'm just as entitled to an opinion as you are and I would appreciate it if the nixpkgs community was a little less rude to anyone who disagrees with some dogma no matter their knowledge. The nixpkgs community is not regarded as healthy or friendly after an internecine faction war that split three ways twice, producing four implementations none of which work, and if that reputation is ever going to mend, it will be because less than every Nix person has precisely zero chill the nanosecond anyone disagrees with them about anything.
It's no different other than it's a layer of indirection which allows bash to change in a way that would not be possible if scripts directly referenced /bin/bash or /usr/bin/bash. The only thing you can't change is the /usr/bin/env binary itself but that's not something that changes very much if at all. On my nix box, bash is at /run/current-system/sw/bin/bash and "current-system" is a symlink to a particular build in /nix/store. So /usr/bin/env is creating a layer of indirection so if I rebuild the system, the previous version can be kept around and I can roll back instantly at the boot menu and it will roll everything back to a consistent package set. Now as I say, you could achieve this in other ways eg fs snapshots etc. But to say that the change doesn't give you anything is just very obviously wrong.
And you don't need to worry- it's perfectly obvious how deep your Nix expertise is.
You've just used a personal attack to shout down someone who disagrees in good faith over a distinction you simultaneously describe in your own words as "something you could achieve other ways" (yeah, more than a little bit) referencing an artifact that "is a symlink", "on my nix box".
The only thing I attacked are named instances of regrettable community malfunctions stripped of anything personal to an individual that hurt both current users of Nix and people who might experiment with systems they can reason about, saw an interaction like this, and did something less painful with their day. Because my evident and sincere concern for the future of Nix is framed as stark disagreement over a few particularly sacred cows.
At least on `comp.lang.lisp` you got your middle finger alongside a worthwhile education:
"There are some things in life that you do not do if you want to be a moral being and feel proud of what you have accomplished."
My reaction was negative when I read the title (policy!), but the actual outcome was very reasonable. "Hey, can we add $ORIGIN to the VFS layer to support relocatable interpreters?" "You can already do that with binfmt_misc and ebpf, here's an example."
Then, later, a couple of patches to make binfmt_misc more ergonomic. Seems like a good outcome.
Nice, PT_INTERP is the only non-relocatable thing of ELF files and typically requires wrapper scripts/executables.
Regarding shebangs, I've never understood why the kernel cannot resolve e.g. `#!sh` relative to PATH instead of CWD. Posix prescribes that you should look for `sh` in PATH and don't expect it to be in `/bin/sh`. And using `/usr/bin/env sh` has the same issue: what if coreutils is installed elsewhere.
I suspect it's a mix of historical reasons (#! support was added pretty early, in 1980 or so, when the unix development philosophy I think tended quite strongly to "do the simple thing", and there wasn't so much variation in where you might put important binaries like the shell), plus the fact that the kernel doesn't know anything about PATH (it's only your shell that does), plus vague worries about potentially accidentally breaking existing #! lines that used to work.
> plus the fact that the kernel doesn't know anything about PATH (it's only your shell that does)
glibc is what deals with PATH, not the shell. On Linux, it's execvp(3) in libc which is implemented in terms of execve(2) in the kernel, but POSIX doesn't make a distinction between syscalls and library functions, so a kernel could implement either or none directly.
A surprising number of things in Linux are transparent to the kernel as they are implemented in user space. For instance, the Linux kernel doesn't support multithreading. It only supports processes sharing an address space. Which is what most people call threads - but in the kernel, they are just processes that happen to share an address space.
> the Linux kernel doesn't support multithreading. It only supports processes sharing an address space. Which is what most people call threads - but in the kernel, they are just processes that happen to share an address space.
This isn’t really true. It is just that internally, the kernel calls the thread ID “pid” and the process ID “tgid” (“thread group ID”). But the getpid system call returns the “tgid” not the “pid”, and to get the “pid” you need to call gettid (get thread ID). Other kernel interfaces also use the terminology “thread”, e.g. /proc/thread_self, set_tid_address, set_thread_area/get_thread_area, the CLONE_THREAD flag to clone, inter alia
I found grandparent's post still useful because it caused something to click in my head. I then researched and found that threads and processes share a lot of functionality. In the kernel both processes and threads are represented by `task_struct` instances.
So, respectfully, you shouldn't dismiss grandparent with "This isn't really true".
This is really awesome, can't wait for this to land in the linux kernel.
I have made appseed[0] (asciinema link[1]) which used zapps[2] which did a lot of tricks to achieve in end what this kernel patch could help in doing and hopefully more in the future!
My appseed project didn't work for large projects like OBS but I feel like the approach that this does could lead to mass adoption given that its now a kernel feature. It could in theory allow more portable binaries across Linux which is really cool :)
Many people here seem to think this change will allow for people to package their Linux apps to be portable across distros/distro releases. If this is true then this a great!
On Debian-based distros it kind of sucks when the version of a package is only available in a newer release so the only way to get it is either compile it yourself or upgrade your system.
I remember trying to hack around this once by downloading dependencies from a newer release and helping the program locate them but I think I ran into a issue with the linker version being hardcoded or something and gave up.
>Many people here seem to think this change will allow for people to package their Linux apps to be portable across distros/distro releases.
There is no need to make binaries portable across the distros, because Linux is a source based OS. You can always just compile anything for your own system, just don't forget to submit the buildscript to the repo.
I wonder if something equivalent exists for loading other kinds of assets. Take files that would normally live under `/usr/share`. References to these files are typically hardcoded with an absolute path as well. It's usually possible to relocate them at build time by specifying a different absolute prefix, but making them relative the conventional way would likely require patching.
Sure, assuming this is your own software. The context of the article is on working towards a general approach to relocatable nix binaries so I don't think that's going to scale across nixpkgs.
linux ELF code is loading the ELF loader only. $ORIGIN may not be a good idea
since that would add more ELF complexity to the kernel.
If we are honest with ourself, ELF is the core of the issue: for executables and
dynamic libraries we _now_ know it is severely obsolete on modern hardware
architectures.
I am currently using my own format, excrutiatingly simple, no loader, basically a
program segment, "userland syscalls" with hardware CPU synchronization. I do wrap
the executables into ELF capsules to run them transparently. So simple a small RFC will be enough.
With that, I discovered that the hard part is c++ and other similar languages which are very expensive in runtime infrastructure and linking complexity. I
would need to build a mesa vulkan driver with that format, and it seems the blocker is c++ (and similar language namely with grotesque and absurd syntax
complexity). Thx to valve to have removed a lot of c++, for less c++... would
have been much better if plain and simple C.
I would love to learn more about this;
I'm deeply interested in replacing ELF with something either simpler or maybe leveraging sqlite for much of the functionality.
Once I have something satisfying, I'll post the "RFC-like" draft and preliminary reference toolchain on HN (well, if I can since HN is going 'whatwg cartel' web engines only then I may be blocked for good). For the moment, I am 'testing' all that while writting my own wayland compositor for linux (that's why I will try to build mesa AMD vulkan driver for this format... or stick to wl_shm in the end).
Well, for a new executable and dynamic library format, the computer language syntax complexity and its runtime complexity do matter A LOT, c++ is massive pain, and since, if I am not mistaken, sqlite is going microsoft rust, that's bad omens since microsoft rust syntax seems to be now as brain damaged than c++ syntax... but with an even worse runtime (unless all that is not actually true, I have not checked). That said, ISO f*cked up C with things like '__thread' which requires some level of OS support.
Anything that assumes /proc/self/exe points at the real binary, sandboxes and supervisors especially, quietly breaks under this. Worth a grep across that class of code before it lands broadly.
Its unfortunate that this is using eBPF and not cBPF. I'd want to enable this in a namespace using binfmt-misc's namespace support, but eBPF requires root.
Nix has caused so much busywork to be created through the entire Linux ecosystem. Rather than fixing their own project they force work upon so many others.
Instead of looking for ways Nix could work with ensuring else they force everything else to be modified to work with Nix. You will constantly seem them trying to get projects to adapt a flake.nix or to replace bash scripts with a different shebang since they hardcode a single binary that is rare for people to use in the real world.
>Rather than fixing their own project they force work upon so many others.
Nix has put in a ton of work to make their project work, and the assumptions in the rest of the ecosystem have been fighting every step of the way. And I would argue they've been changing it for the better, considering that if you were to just 'try to make nix work' you'd probably mainly come to the conclusion that it was impossible to make work sensibly due to the huge number of places where hardcoded relationships have been baked in.
Many of those assumptions are standards such as the filesystem hierarchy (granted, that's an xkcd 927 situation), nix is a bit of a special snowflake that breaks things in subtle ways (e.g. [0][1]) and needs workarounds in various places.
Those workarounds are not necessarily onerous, but they would be unnecessary if it didn't deviate.
>and the assumptions in the rest of the ecosystem have been fighting every step of the way
There is an arrogance in the way of thinking that when you see everyone breaking your arbitrary rules you think to yourself "there is no way my rules are at fault, it's everyone else's fault for breaking them."
>due to the huge number of places where hardcoded relationships have been baked in.
And why are hardcoded relationships bad for developers or users of the operating system? "Because they break Nix" is not a good excuse. Plenty of other operating systems work just fine with such things hardcoded, I don't see developers complaining about these hardcoded things, and moving things around can only make things more confusing for the user.
To me it looks like Nix picked a poor design and instead of going back to the drawing board and trying to come up with a better design they have gone all in on trying to force their bad design to work.
How much work do you think iOS or Android spend on app packaging / installation per year. Despite having millions of more developers and billions of more users they spend orders of magnitude less than NixOS does.
> There is an arrogance in the way of thinking that when you see everyone breaking your arbitrary rules you think to yourself "there is no way my rules are at fault, it's everyone else's fault for breaking them."
It's not really arrogance - it's just a different way to design a system. NixOS wants a few things:
1. An input-addressed dependency system
2. Easy and robust rollback/upgrades
3. Different versions of software/libraries/services coexisting on a single machine.
A lot of this is just impossible to have together with the way software were written, so NixOS made the changes necessary for them. When it became obvious it might be useful to other people, they upstreamed it.
I do agree with you that the whole `#!/usr/bin/env bash` shebang shenanigan is kinda ass (my personal nixos machine has a `/bin/bash` alias because there's not enough time in my life to care about this issue). But I also think that NixOS is right here, shebangs requiring a full path is just a terrible design in a multi-user world. If different users want to have a different bash implementation, they should be able to!
---
Fundamentally, I think the NixOS OS design is actually beautiful. It's taking most of what makes modern phone OS so reliable (read-only system, A/B partitions, etc...), and bringing their ideas in a shape that is coherent with desktop/server OS design.
The correct fix is to store the metadata outside the CAS, in for example, the loader, which can trivially delegate to another loader (does on my machine now).
bazel's wrong solution is $ORIGIN, Nix's wrong solution is "floating" CA / "realizations".
The technical debt / precedent argument doesn't apply: Nix never had this bug (for once), you the choices on this bug are 1. port the bazel bug to Nix 2. don't port the bug.
I predict... the bug is a shoe in, the bug lands eight days a week.
https://fzakaria.com/2026/06/21/nix-needs-relocatable-binari...
> The loader in Linux however natively supports the variable $ORIGIN which translates to “the directory containing the executable.”
https://man7.org/linux/man-pages/man8/ld.so.8.html
But, if ld.so supports $ORIGIN already, why does the kernel needs to support it also? Or rather, why can't the kernel leverage ld.so and do this entirely on userspace?