[remram]

I think the problem of systemd is that it is complex. It is fine when it works correctly, but when you eventually have to dig into any sort of problem, it is hell.

For example, all of my boxes spew many logs per minute about "Failed to set up mount unit". This is apparently a bug with the generated name of some internal unit related to mountpoints (too long). This one is not as bad as others, because it does have a bug reported for it, but is still something I'll have to deal with until I upgrade the distro on all the servers. Many similar bugs I can't track down at all.

Systemd makes a lot of things easy, but not simple, and that is a big problem in practice.

[twic]

This is it. For example, systemd (a) allows you to run services as a normal user, not as root, (b) treats child processes of a service process as part of that service, and (c) collects logs from services. Except if you try to do all three, that just doesn't work, because they've half-arsed the log collection process:

https://serverfault.com/a/1143851/92104

There's no reason for this not to work! Taking a very straightforward and traditional approach to running processes (as in my demo code in that question), you don't have this problem. But rather than that, they did something clever which doesn't work properly.

[thwarted]

I can't make heads or tails of either of those answers, and they seem to contradict each other. The output capturing is all to the same file descriptors, this is the way resources are inherited on fork (and there's no redirection here). That it works with unbuffer is weird because the subshell and subprocess both end up flushing their output when they exit, so there should be no need to flush before exiting.

If the first answer is correct, then the unbuffer works because the process introspection as the source of the output is unified to be the unbuffer process. But why bother introspecting the exact process that originated a write to the output? Did we have a problem with file descriptors being highjacked? And even if we did, losing the output (or not associating it with the parent process) is worse.

[twic]

AIUI, what happens is that no logging is ever lost, it is just not associated with the right service. This happens when a child process spawns, writes something, then dies before systemd is able to find the child's PID and map it to a service.

This implies that systemd does not just set up a pipe for each service and read everything from that. Rather, it does something like set up a unix domain socket, and then use SO_PEERCRED to find out who is at the other end for each write, then map that to a service by walking the process hierarchy or something. Maybe it's something less nuts than that, but i am not optimistic.

[thwarted]

Yeah, that's what I gathered too. But if you can't find the output, output that is implicitly associated with a process tree as in this example, it is effectively lost. And even if you could find it, that it's disassociated from the origin reduces the utility of having captured the output at all.

SO_PEERCRED is on a per-socket basis, not per-write basis. From unix(7):

              This read-only socket option returns the credentials of
              the peer process connected to this socket.  The returned
              credentials are those that were in effect at the time of
              the call to connect(2) or socketpair(2).

Child processes that inherit the file descriptor like in the example would not influence the SO_PEERCRED result. So it must be something else. For vague reasons.