[pdenton]

It's pretty cool actually that this is possible and I also can't help but think that this is a solution to a problem that need not exist in the first place. Systemd has this problem and the solution is to use a feature in systemd, right?

IMHO systemd has amassed too many features, both documented and undocumented. Having worked with GNU+Linux for over 2 decades, I still find Devuan and Void easier to manage than Fedora and Ubuntu. The latter is fine for basic desktop needs though.

[jraph]

> Systemd has this problem

It seems like it is a solution for anything that lives in /usr and could break your system / cannot directly be put there because it's read only when you are developing it. Not systemd-specific though obviously the part that manages the system boot and system services is going to be particularly concerned by this.

> IMHO systemd has amassed too many features

But systemd is not some monolith that does a lot of things. It's a set of tools that do one thing, well, quite like GNU and that happen to be called systemd-something. Often, thin wrappers around kernel features, like nspawn. I find them very cleanly designed and easy to use. To each there own but I find the possibility to create services declaratively with systemd unit files particularly useful, instead of having to write custom shell scripts and copy paste boilerplate. Upstart was a step in the right direction in this respect. It's not just "fine for basic desktop needs", it's a great fit for desktops and also particularly helpful on servers as well (not speaking about Ubuntu, just systemd).

[ldng]

"But systemd is not some monolith"

It. Does. Not. Matter.

Still to much code in one place. Not enough review and unrelated subproject bitrot at different rate without visibility.

[lazyier]

That is like saying Apache project is subject to not enough review and unrelated subproject bitrot.

https://projects.apache.org/projects.html

or GNU

https://directory.fsf.org/wiki/GNU

Or FreeBSD or OpenBSD....

Having a single large overall project that manages a diverse number of individual software products is a common pattern and certainly something that isn't remotely unique to Systemd. There is no reason to denigrate systemd for that.

[ldng]

You're making an Apple to Orange comparison here. Your examples are not released to the outside world as a single product "take-it-or-leave-it" and painting it so is borderline bad faith.

[throwaway82652]

Painting systemd in that way is also bad faith, because it's not a single product "take-it-or-leave-it" either. If you think it doesn't have enough review for your purposes then it's time for you to step up and start reviewing, either in systemd or in another project.

Part of the reason why people seem to (falsely) think systemd is monolithic is because distros have been historically bad at packaging it and haven't split up the components, but that's been improving over time.

[ephaeton]

I share your opinion, but IMO it also goes one step further: systemd uses its gravity to try and shape linux distributions and dictate how things "ought" to work. Consider this quote from the linked article:

> Oh, and in case you wonder, all of this only works with distributions that have completed the /usr/ merge. On legacy distributions that didn't do that and still place parts of /usr/ all over the hierarchy the above won't work, since merging /usr/ trees via overlayfs is pretty pointess if the OS is not hermetic in /usr/.)

IOW, if I build a system with yocto for an embedded system, which is also a very useful candidate for a read-only system image installation, I cannot use this feature for this use-case because ... yeah because what. Because lennart says so. This got old so fast.

[jraph]

That's fair enough though, isn't it?

We can't blame the developers to not support any oddity and particularity that exists… and then blame them because the distribution makers adopt their tools because they work well (as a result?).

I'm happy they focus on a standardized base, this results in less development effort focused on more useful stuff, cleaner and more robust tools, it also makes it easy to switch between distributions and to target this standardized base when developing.

It also can be seen as a assumed limitation. Any tool has limitations. But the /usr merge has been conducted with reasons, and makes this kind of stuff actually possible. It's not like keeping a split /usr is very useful today anyway.

Systemd is not something distributions have to bend to. It's something they willingly adopted because it solves well problems they have.

And it's not just Lennart. Hate he gets gets old fast.

[kasabali]

> We can't blame the developers to not support any oddity and particularity that exists

It isn't as innocent as you paint it.

systemd developers have other agendas when making decisions , you can hear it from horses mouth:

"Well, it is definitely our intention to gently push the distributions in the same direction so that they stop supporting deviating solutions for these things where there's really no point at all in doing so." [1]

And after watching systemd development for years I might argue they have a very liberal interpretation of what constitutes "senseless configuration differences between the distros for the really basic stuff"

[1] https://lists.freedesktop.org/archives/systemd-devel/2010-Se...

[lazyier]

Unified low-level "linux plumbing" architecture is something that a lot of people have realized is needed for a very very long time.

This is a major criticism against Linux distributions from the beginning. Especially from the *BSD camps.

And Systemd project is full of people actually putting the work into solving this major problem.

You are right that it isn't innocent. It is actually very sophisticated and probably very wise thing to do. It is helping a lot of people out.

[kasabali]

What's a "major problem" to you is the best and unique aspect of having different linux distributions to others.

The actual major problem is that systemd developers wasn't being contempt with just building an opinionated system and let it rise on its merits, instead they choose to use its status to "gently push" their opinions on everyone else.

[throwaway82652]

If your distribution is already content to ignore any efforts at standardization in favor of "unique aspects" then I don't see what the problem is. You can go about your business and ignore systemd, or choose to adopt it on its merits. It's entirely up to you, nobody is using any status to push opinions on anyone.

Also you're ignoring the last part of that sentence: "where there's really no point at all in doing so". That's the most important part. If a distro has a real reason to deviate then that's fine. If a distro is deviating for no reason then that's doing a disservice to its users, and you can expect anyone who ships other things for that distro to be upset with them for pointlessly making extra work for everyone else. It's not just the systemd developers who will be inconvenienced.

[ephaeton]

There's no need to be politic about this from the POV of systemd. "We support this feature on merged /usr" does not have to mean "we don't allow to run it on legacy systems because it doesn't make sense". Look, your users have more fantasy than you do how to use your software. Doesn't mean you have to support it. But it also doesn't mean you have to explicitly enable it only in the setup you envisioned this being used.

It's not an _assumed_ limitation. It's an _enforced_ limitation based on a limited world-view that also uses the gravity of the project to enforce the world-view on any user.

I'm not arguing pro or against split /usr. Reality though is that for various reasons there are legitimate use-cases for a non-unified directory tree, e.g. embedded ramdisks that contain only the barest of /bin in a crunchgen'd binary that don't want to duplicate stuff that's later used in /usr/bin. Linux is not only desktop and server.

systemd is being adopted because it solves problems well the distributors have, but THEN the distributor ALSO has to bend their distribution to the single enforced world-view the systemd folks have if they want it to integrate. They, the distributors, software integrators per se, are just forced to hack out limitations that are imposed by systemd folks for no benefit of either; systemd dev or user. No unified-/usr-user will benefit from this enforced restriction. A split-/usr-user will have to patch systemd and maintain their tree to use this feature if they have a good use-case for it because .. "systemd says so" (if you prefer that over the "B"DFL lennart).

Lastly, you are correct in that systemd is a pool of like-minded developers - not only lennart; and this is where I add: - who like to shape the reality out there instead of allowing distributors and users to easily take on their own responsibility easily. Sure, I can fork systemd (it's free software after all), but it's unnecessarily "opinionated" for its broad area of application. And that's where the "politic" dimension of systemd development comes into play ...

[jraph]

> There's no need to be politic about this from the POV of systemd

I don't think anybody is. It's their technical approach to develop stuff. But sure, their approach has a huge impact on the ecosystem.

> we don't allow to run it on legacy systems because it doesn't make sense

If I'm to design some piece of software that requires something, that this something is still relatively commonly not there and that I know it won't work without, yes, I'll go ahead, make a check and prevent the program to run in this configuration.

Think ./configure which will not write the Makefile if some critical dependency is missing. It's the same thing for me.

Anyway, the author did say "it won't work", he didn't say "We went out of our way to prevent the users from trying to do this if we detect an unmerged /usr". I think you read too much into the whole thing.

[ephaeton]

> But sure, their approach has a huge impact on the ecosystem.

And this is the political dimension of systemd. One they keep using to shape the ecosystem to their preference.

> If I'm to design some piece of software that requires something,

See that's the point. It doesn't _require_ a merged /usr to work. The benefit of the merged /usr here is that this guarantees to potentially overwrite/extend "all" binaries, not "only" those in a split-/usr. The latter doesn't mean it doesn't work. It just means that they only overwrite one place, not all potential places something exists.

[jraph]

> > But sure, their approach has a huge impact on the ecosystem.

> And this is the political dimension of systemd. One they keep using to shape the ecosystem to their preference.

Any highly successful project is political then. Linux developers in particular force their kernel and their views on how a kernel should behave and be designed down the throats of their users.

... Obviously you have your say when people choose to use your stuff. But then that's fine, you are the expert in your domain. Who else is better placed than you? If people use your stuff, probably preferences are shared.

Or they are locked-in (through network effect for instance). But the systemd folks are not out there having an evil agenda they force the ecosystem into. They solve their problems and people do happen to adopt their stuff. They do seem to have a very "if it's broken, fix it" approach but that seems to work out quite well. Sure that's not always comfortable but I find it somewhat refreshing.

[lazyier]

> systemd is being adopted because it solves problems well the distributors have, but THEN the distributor ALSO has to bend their distribution to the single enforced world-view the systemd folks have if they want it to integrate.

It is like saying that "It is great that engineers are working to provide a unified rail line across Europe, but they are assholes because they want everybody to use the same rail gauge as well as implementing these changes that will allow the rails to handle traffic going 200 mph. The unified rail folks have an agenda!!"

The problem they are trying to solve is solved by "bending of the will", so to say.

[mtzet]

> IOW, if I build a system with yocto for an embedded system, which is also a very useful candidate for a read-only system image installation, I cannot use this feature for this use-case because ...

I agree that this is precisely a nice use-case, but why is this a problem? I agree with Lennarts reasoning that it makes little sense if the files are arbitrarily split between / and /usr. In any case, Yocto has had support for usrmerge for a while now[1].

[1] https://docs.yoctoproject.org/ref-manual/features.html?highl...

[ephaeton]

thanks for the pointer!

[throwaway82652]

>IOW, if I build a system with yocto for an embedded system, which is also a very useful candidate for a read-only system image installation, I cannot use this feature for this use-case because ... yeah because what. Because lennart says so.

No this is wrong. You can still use this feature but you get to manually type out all the mount paths yourself every time and work out all the bugs in it when you try to mount other things on top of it, because that's what you signed up for when you decided to have multiple paths. I'm really baffled as to how you could even remotely say this is a systemd problem. This will be a problem if you try to do any kind of bind mount or overlayfs stuff on that system.

I really suggest that you avoid making comments like "systemd uses its gravity to try and shape linux distributions and dictate how things 'ought' to work". That stuff is conspiracy theory territory at this point, it's so disconnected from reality that it's just pointless to say it.

[mtzet]

> Systemd has this problem

How does systemd have a problem? Systemd doesn't care about having a read-only rootfs at all, except that it supports it and now ships a little tool that's useful if you happen to use them. Fedora and Ubuntu doesn't ship a read-only rootfs(1).

(1) Well, Fedora Silverblue is an experimental Fedora-variant that uses a read only rootfs. But that point still stands.

[cookiengineer]

I am not sure whether or not you are aware that some might intepret your comment as survivor's bias. Not everybody is a sysadmin, and definitely not everybody wants to do those tedious tasks in /etc.

There are so many potential pitfalls of everything around init/initd, especially if downstream integration scripts have to keep up with upstream changes.

And I would argue that those should not be the burden of endusers, and also not be the burden of distro maintainers. Providing a settings file that can autolaunch your application in a failsafe manner for all distributions is where systemd shines at, like it or not.

While I agree that it isn't perfect and the debugging critiques are kind of valid, I also would argue that this can be solved with better (graphical) tools for the systemd ecosystem.

Also: if you assume that debugging a broken shell script (running as root) is easier than systemd, then your perspective is already biased, cause it's a skill set that requires lots of experience.

I mean, most shell scripts in /etc/init.d don't even set -e, let alone -u ... and should be considered a risk for system integrity.

[vegai_]

>Systemd has this problem and the solution is to use a feature in systemd, right?

More like systems with an immutable /usr have this problem, and systemd now has a way to circumvent those restrictions when developing. Fedora has been prototyping such systems for quite some time, not exactly with brilliant success so far, but it's good that they're trying.