[jstimpfle]

If you look around there are lots and lots of objects that are "distributed", or aren't but should be. Synchronicity is often what's killing performance and introducing complexity.

> You surely don’t want to continue reading the remaining 3.9GB source file and call write() in noop-mode another thousand times in your loop before realizing this error on flush

It can be completely reasonable to back out only at strategic points. Copying a few KB or MB of memory more will rarely matter for an error case that shouldn't be optimized for. If there is an error, you'll typically want to reset a larger context object anyway. It depends on the situation, but by not having to handle the error at first notice, you can sometimes simplify the logic.

> What do you do when write() fails because the buffer is full (EAGAIN)?

EAGAIN is a different beast, it's not a "real" I/O error. With better APIs you retrieve buffers first (often in a different phase), removing this class of errors completely. But you can mostly just ignore EAGAIN anyway. It's a transient error (or not an error at all, really) that simply tells you the reason why zero bytes were written.

With fwrite(), not sure if it is well specified how it should interact with non-blocking FDs and EAGAIN. Probably it doesn't even allow you to distinguish between EAGAIN and I/O errors. It could also be an option to return a short write in this case (but I believe fwrite() needs to set either the error of EOF flag if it returns a short write). I also think fwrite() is largely not used with non-blocking FDs.