[eeks]

It is limited to UDP for the moment (at least AFAICT). I would be very surprised if they ever offer user-space TCP. It's a much different beast than UDP to handle right in user-space.

Also, the 30% less overhead is underwhelming. I would have expected much better improvement than that. That being said, if their measurement includes the encryption layer then the I/O benefit may be overshadowed by the encryption overhead.

Still, pretty cool stuff to make publicly available.

[eridius]

It definitely offers TCP as well. I had an Apple engineer confirm to me that the kernel still provides sanity-checking of the packets and implied that the kernel will still terminate a TCP connection if the process crashes, both of which only make sense if the user-space networking layer is handling TCP.

[eeks]

This implementation looks a lot like HW TCP offload engines where the kernel handles session creation and termination and the HW takes care of most of the state machine. Apple then must have found some way to hide the handling of ancillary tasks from the user in a way that does not cripple the protocol. They may have added hooks in the main application loop but my guess is that they are running separate threads to avoid having the application hanging the main thread in a non-returning loop and prevent these ancillary tasks from running (some are time sensitive). This means that their TCP user-space session management is most likely multi-threaded, which has a detrimental impact on performance due to the use of locks and the consequential cache pollution.

[eridius]

In the WWDC session on this, they demonstrated a simple app that sent uncompressed video frames captured from the camera over the network (I think using TCP, but I don't recall for sure) using BSD sockets and using Network.framework and reported 30% less overhead with Network.framework.

Which is to say, you're saying "detrimental impact on performance" and yet this seems to be a significant win over BSD sockets.

[eeks]

> I think using TCP, but I don't recall for sure

It was UDP.

> Which is to say, you're saying "detrimental impact on performance" and yet this seems to be a significant win over BSD sockets.

The point I was trying to drive home is that, while 30% overhead reduction compared to BSD socket is nothing to laugh at, a fully user-space UDP/TCP network stack combined with memory-mapped buffer sharing usually gives performance improvement measured in the "x" and not in the "%".

Now, there is not much in terms of experiment protocol in their material so its very hard to tell. You mention uncompressed video so they could be sending humongous frames, which I doubt as no one ever would send raw frames over the network (that would be several MB per frame for a 720p front camera). But if that is the case then the data copy operation becomes the predominant bottleneck and getting rid of one may justify the improvement. But that is not a realistic scenario.

The more realistic scenario is that they sent compressed delta-frames across the network (H264 or HEIF), which would then considerably reduce the transferred payload size. In that scenario, data copy is not the predominant overhead anymore and 30% overhead reduction is underwhelming, telling me that they still are calling expensive operations like syscalls/uIPC on the critical data path.

[eridius]

According to what they said in the session, they were asking for video frames from the camera and sending them, completely un-interpreted, over the network. The idea being the receiving device could take these frames and handle them exactly the same way as they would handle data coming from the local camera.

[doctorsher]

According to their slides from WWDC last year, it also offers TCP in userspace.

[eeks]

If you are referring to "Advances in Networking, part 1" [0], the material used is a little thin to draw any conclusion regarding their user-space TCP/IP design.

[0] https://devstreaming-cdn.apple.com/videos/wwdc/2017/707h2gkb...

[doctorsher]

That is the talk I was referring to -- and I do agree the material is thin. But here's what I was basing it off of: the userspace networking slide has the "TCP / IPv6" block in userspace. While it's true this could be a broad block (it doesn't include UDP for example, which we know is in there), I doubt they would have explicitly put TCP if it was not true.

[eeks]

> I doubt they would have explicitly put TCP if it was not true

I am not doubting they have some sort of user-space TCP implementation. I am trying to understand how much of TCP they moved out of the kernel. From what I can gather they still have enough of it in the kernel such that the related syscall/uIPC overhead does not allow more than the 30% performance improvement announced.