[dotancohen]

If you're bringing that up, then this is the place to spread the word about my dream of a stdmeta file descriptor.

https://unix.stackexchange.com/questions/197809/propose-addi...

Just like we have stdout and stderr, header lines such as those produced by `ps` should be printed to stdmeta. Curl is the worse offender here, outputing meta lines to stderr instead of stdout. A stdmeta file descriptor would make it clear what is data, what is an error, and what is _describing_ the data.

[runlevel1]

That reminds me of something I've wanted for quite a while:

A ringbuffer filetype. Similar to a named pipe file (see: fifo(7)[^1]), but without consuming the contents on read and automatically rotating out the oldest lines.

Of course, there would be some complexities around handling read position as lines are being rotated out from under you.

[1]: https://linux.die.net/man/7/fifo

[naikrovek]

> Of course, there would be some complexities around handling read position as lines are being rotated out from under you.

that seems solvable to me, but multiple simultaneous readers... that seems like it might be a bit more challenging...

[speed_spread]

Only allow sequential reads. Each reader gets it's own cursor. If writer catches up to slowest reader (buffer full) it bumps it up before it writes. Clients can query current cursor position to know if they've been bumped. Or something like that.

[anonymoushn]

If we're adding standard output fds, maybe it would be a useful time to define any mechanism for consumers of those fds to discern the total ordering of bytes written to the three of them.

[dotancohen]

WC has a "character" flag that really just counts bytes:

  $ echo דותן | wc -c
  9

Note that each letter is two bytes, and the newline is an additional byte. So you could pipe (or tree) to wc to count bytes. For a hypothetical stdmeta on fd 3, that might look like this (piping stdout to devnull):

  $ foo 3>&1 > /dev/null | wc -c

[anonymoushn]

I'm not sure what you are answering. I am writing about the thing where some programs intentionally write different byte streams to stderr and stdout that will be interleaved a certain way if stdout and stderr turn out the be the same thing, but if the user wishes to distinguish which bytes were written to which stream, they can no longer recover the correct interleaving of the bytes.

[lstamour]

The best compromise I can think of would be to prepend the line source to the combined output, so lines in the combined output can be distinguished. But depending on how you do that, you might have timing issues. In fact, any form of interleaving the outputs of stderr and stdout are technically subject to timing issues because even the source application can’t specify that bytes in one stream should follow bytes in another stream. https://stackoverflow.com/questions/12517519/how-to-redirect... talks a bit about the problem, and I’m sure someone somewhere has written a general-purpose tool, but there are timing edge cases here and often the solution is seen as keeping errors messages exclusively in stdout (or logs) or more practically, not worrying about preserving order and writing errors exclusively to stderr without worrying as much about strictly when errors interleave. If you add timestamps to your line outputs, or add some form of line numbers to the output, you can restore order after logging but it would be application-specific to do so. From a practical perspective, listening to both streams and quickly appending to a list with a data structure containing the source and message text is probably your closest approximation if writing code to handle output streams and you want an combined output that can still source which stream came from which source. (Again subject to timing errors)

[WorldMaker]

You could require orderable sequence numbers before or after each line. It would be easiest as yet another VT escape sequence, but if you are pie-in-sky adding more standard "files" anyway maybe you wish for lines to be newline-delimited JSON CRDTs while you are at it.

[dotancohen]

Yeah, I see now what you wrote. My coffeeless brain must have just seen "total ... of bytes". Carry on...

[jwilk]

For the avoidance of doubt, it's not a bug that -c counts bytes, not characters.

There's a separate option for counting characters: -m

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/w...

[rjh29]

Any metadata mechanism should be extensible IMO. For example sort and join care about ordering but they also care about column names.