Hacker News new | ask | show | jobs
by rkerno 431 days ago
It's pretty easy in a push based model to let the 'pusher' know that no more data is required. It's just like unsubscribing from an event, or returning a 'no more' status from the callback. The push model does feel more natural to me, but perhaps that comes from familiarity with linux piping.
1 comments
geocar 431 days ago
It's easy when the network is working.

If it isn't, the 'pusher' continues to fill memory buffers that can take minutes to dequeue. You need constant communication and TCP conspires against you on this. If your flow is primarily one-directional, you might be able to use keep-alives for this, but the defaults are terrible. Here's what I have used:

    SO_KEEPALIVE=1
    TCP_KEEPCNT=1+x(seconds)
    TCP_KEEPIDLE=1
    TCP_KEEPINTVL=1
    TCP_NODELAY=1
    TCP_USER_TIMEOUT=x(msec)
where 'x' is the anticipated RTT of ping+connect (usually 3-4x the measured RTT you get from TCP_INFO).

Remember: the moon is only about 1.5 seconds away, so unless you're pushing data to the moon these numbers are likely very small.

On older Windows, you've got `SIO_KEEPALIVE_VALS` which is hardcoded at 10 ticks, so you need to divide your distance by 10, but I think new windows supports `TCP_USER_TIMEOUT` like Linux.

Mac doesn't have these socket options. I think you can set net.inet.tcp.keep* with sysctl, but this affects/breaks everything, so I don't recommend xnu as a consumer to a high-volume push-stream.

I actually don't recommend any of this unless you have literally no control over higher-level parts of the protocol: TCP just isn't good enough for high-volume push-protocols, and so I haven't used this in my own stuff for decades now.

link
hinkley 429 days ago
It was kind of a toy problem, but I had a lot of fun with consul’s versioning data and making requests that didn’t return until the data had changed. It’s like GetIfModified but you can set it up with a delay if the data has not already been modified.

I don’t think it’s particularly good for multi-get but I didn’t get that far.

There are systems where you can report what you’ve seen so far when you reconnect, but those require a very different architecture on the sending end because it’s not just a streaming data situation at that point. And while it’s more efficient to store the data to derive the replay than to store the replay itself, it’s not infinitely moreso and a three hour network hardware problem could really ruin your entire week.

link