[kennylevinsen]

This argument does not make sense - the kernel already needs to track per-process file descriptors. It just looks for the first hole instead of giving the "next" value.

Go's random map iteration does not apply here. Not only is this not an iterable map, the kernel has no problem providing this insertion guarantee so adding additional costly randomization has no benefit and just burns additional cycles.

Go would also be better off without, but they are catering to a different audience and different degree of specification, and apparently need to actively deter developers from ignoring documentation.

[infogulch]

The correct term for this is not "developers ignoring documentation" it's "ossification" or Hyrum's Law:

    With a sufficient number of users of an API,
    it does not matter what you promise in the contract:
    all observable behaviors of your system
    will be depended on by somebody.

I guess that we got this "lowest available" rule because that's what the first implementation happened to do (it's the obvious thing to do if you have a single core), then someone 'clever' noticed that they could save 3 cycles by hard coding and reusing the fd in their IO-bound loop, and anyone that tried to implement fd allocation differently was instantly met by "your OS breaks my app", and thus the first implementation was permanently ossified in stone. To be clear I'm not making any historical claims and this is pure speculation.

"Stupid developers should have rtfm humph" is not a useful position because it ignores this behavior ossification.

The Go map example is actually very relevant, it's an "anti-ossification" feature that makes the behavior match the spec. If the spec says iteration order is not guaranteed, but in practice people can rely on it being the same in some specific situation (say, in a unit test on a particular version of Go) then the spec is ignored and it breaks people's programs when the situation changes (e.g. Go version updates). This actually happened. Instead of giving in and ossifying the first implementation's details into the spec, Go chose the only other approach: Make the behavior match the spec: "iteration order is not guaranteed" == "iteration order is explicitly randomized". (They do it pretty efficiently actually.)

[kennylevinsen]

As mentioned elsewhere, the file descriptor table is an array and a bitmask - finding the next fd is a matter of finding the first unset bit, which is extremely efficient. And that's before we ignore that the file descriptor table is read-heavy, not write-heavy.

Should you want to have per-process file descriptor tables, you can do just that: Just create a process without CLONE_FILES. You can still maintain other thread-like behaviors if you want. I doubt you'll ever sit with a profile that shows fd allocation as main culprit however.

> If the spec says iteration order is not guaranteed, but in practice people can rely on it being the same in some specific situation ... This actually happened.

If Hyrum's law held, the API would already be "ossified" at this point.

Instead, the Go developers decided to make a statement: "The language spec rather than implementation is authoritative". They broke this misuse permanently by making the API actively hostile, not by making it "match the spec" as it already did.

While one could interpret the current implementation as "anti-ossification", I interpret the action as anti-Hyrum's Law by choosing to break existing users in the name of the contract.

[touisteur]

There you can queue the 'workflow' xkcd https://xkcd.com/1172 and while the joke is funny I wish everyone would stop breaking my workflow.

Maybe I'm getting old, or maybe I find the permanent useless change tiring. I'm looking at GNOME, Android, Windows in particular.

[loeg]

If we ignore POSIX for a moment, the kernel could avoid contending on the one-per-process fd map by sharding the integers into distinct allocation ranges per thread. This would eliminate a source of contention between threads.

In addition to violating POSIX’ lowest hole rule, it would break select(2) (more than it’s already broken).

[kennylevinsen]

This sounds like premature optimization. FD availability is tracked in a bitmask, and finding the next available slot is a matter of scanning for the first unset bit under a spinlock. This is going to be extremely fast.

While you could shard the file descriptor tables for CLONE_FILES processes such as threads, you would likely complicate file descriptor table management and harm the much more important read performance (which is currently just a plain array index and pretty hard to beat).

You could also juts create your processes (or threads) without CLONE_FILES so that they get their own file descriptor table. ------

The fdtable can be seen here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin..., and alloc_fd and __fget can be found here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin....