[twic]

I use websockets quite a lot, for real-time dashboard kind of purposes.

The one thing i really wish websockets had is some kind of application-level acknowledgement or backpressure.

At the server end, you're blasting out messages to the client, but you have no idea if it is keeping up with them. Most of the time, it will be, but if there is a sudden spike of activity, suddenly all your dashboards are going wild, and the client may start to struggle. At that point, you want to be able to shed some load - delay messages a bit, then drop any message which gets superseded (eg if "reactor core temperature is 1050K" is buffered and you get "reactor core temperature is 1100K", you can drop the former). To do that, you need feedback about how far the client has got with processing messages.

You can build a feedback mechanism like this into your application protocol on top of websockets easily enough. But you probably want to do that from the start, or else you will, like me, one day look around and realise that retrofitting it to all your dashboards is a monumental effort.

The RSocket protocol might be a good start - it provides reactive streams semantics, and has a binding to websockets:

https://rsocket.io/guides/rsocket-js

[klabb3]

There should be a law for message passing systems, which says that everyone will eventually want ordered delivery, multiplexing (with priorities), exactly-once semantics, acknowledgements and backpressure. (Maybe more?)

I'm pretty convinced all these popular features could be layered in a reasonable way that could be implemented in most messaging systems, and have standardized semantics and conventions. It seems like every time, we're reinventing the wheel, and half the time people talk over one-another because we're using inexact language.

Basically, what I want is a "message passing a la carte" paper.

[Groxx]

"Every sufficiently-complex system eventually includes a badly-implemented email / lisp / kafka (/zmq/etc)"

Something like a combination of Greenspun's Tenth Rule[1] and Zawinski's Law[2]. Plus whatever would include your queueing system of choice.

Though honestly I've seen more bad queues than emails or lisps. By an order of magnitude or two.

[1]: https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule [2]: https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law

[anderspitman]

I built omnistreams[0] primarily because of the lack of backpressure in browser WebSockets (lots of background information and references in that README). It's what fibridge[1] is built on. We've been using it in production for over 2 years, but I never ended up trying to push omnistreams as a thing. I believe the Rust implementation is actually behind the spec a bit, and the spec itself probably needs some work.

At the end of the day I think RSocket is probably the way to go for most people, though the simplicity of omnistreams is still appealing to me.

EDIT: I just learned about WebSocketStreams[2] from another comment[3] and sounds like they may solve the backpressure issue natively.

[0]: https://github.com/omnistreams/omnistreams-spec

[1]: https://iobio.io/2019/06/12/introducing-fibridge/

[2]: https://web.dev/websocketstream/

[3]: https://news.ycombinator.com/item?id=29894938

[twic]

Nice work on omnistreams!

The WebSocketStream API is a small improvement, because it can leave backed-up messages in the socket buffer, but it still means you're depending on socket buffers for backpressure, which i think is not enough. There's still no way to actually set the receive socket buffer size in the browser, is there?

[anderspitman]

I believe socket backpressure would have worked for my use case. Curious why you think it wouldn't be enough?

As far as I know there's no way to set buffers. IIRC there's a buffer value you can check which is what I tried to use first but don't think that got me very far. Seems like Chrome and Firefox handled it differently or something.

[twic]

As far as i know, browsers do not set a receive buffer size for websockets, so if a client is not reading from the websocket fast enough, the kernel will expand the buffer until it reaches its maximum size, which on my machine is 6 MB. Say you are writing 1 kB/sec of data to this websocket. It will take ~100 seconds to fill.

That means that you won't even know that a client is stuck for a minute and a half (plus however long it takes to fill your send buffer!), and even if you then throttle back, the client has a minute and a half of high-rate data to work through before it catches up. If you throttle up again once you see that the buffer is clearing, and the client gets overloaded again, you will keep hovering around that buffer full state, and the client will keep reading significantly stale data.

To get a useful real-time signal from socket buffers, you need them to be really small. But to get nice smooth transfers of bulk data, you need them to be big, so that is what is the default.

[zamadatix]

If it's streaming data like dashboard statistics then going forward the new WebTransport API might be a much better base: https://github.com/w3c/webtransport/blob/main/explainer.md At this instant it's hot off the assembly line though having just shipped in Chrome 97, Firefox is still working on it.

[rezonant]

Oh wow this is going to be useful

[gfd]

I don't know anything about websockets, but isn't it over tcp? Meaning if the client isn't keeping up, their buffer should be full and the server should be blocked from sending more until it drains (unless it's queuing the messages somewhere else?). Or is that not how tcp backpressure works?

[dmm]

TCP provides backpressure but depending on it to provide backpressure over the internet will greatly increase latency, in my experience.

In one application I was streaming jpeg frames over a websocket and by the time the server application experienced backpressure there were 10s of seconds of messages buffered between the server and client. So the message rate would eventually settle into a rate the connection could sustain but messages would take 10+ seconds to reach the client.

[dragontamer]

Perhaps that sounds like a good time to use TCP_CORK, or TCP_NODELAY flags.

Or, perhaps you need to tune the TCP-Window to your application.

[dmm]

> TCP_CORK, or TCP_NODELAY flags

I'm sending large messages, ~150 kibibytes, so much larger than a typical internet packet. So I'm not sure Nagle's algorithm is the problem.

> tune the TCP-Window to your application.

This is a possibility. I already had to increase wmem_max to handle fast udp connections.

I'll try to put together a minimal test case when I get the chance.

[anderspitman]

One problem is that WebSockets in the browser are 100% asynchronous, ie they don't block on send. So if you have a large amount of data client-side you can easily crash a browser window by sending WS data in a tight loop.

[twic]

It is, but since you don't control the receive buffer size in the browser, or the TCP window size (or, with many web frameworks, the socket buffer size in the server!), you can't rely on socket buffers giving you timely feedback. By the time the server's send buffer fills, there is already masses of stale data buffered in between you and the client.

In my apps i do indeed detect the socket buffers filling, just as you suggest, but pretty much only as a way of detecting completely wedged clients.

[thibauts]

That’s right, but the WebSocket browser API is event driven so the browser HAS TO recv from the socket and dispatch a JS event as soon as data is available.

You’ll get proper back pressure on websockets with synchronous clients that read messages actively.

This is a browser API spec problem more than a protocol problem.

[alienalp]

It is better to implement it at front end. Actually it is not better it is kinda has to be that way. Because page will be already lagging and if you implement it at front at there won't be any lags. And it is better to have limits about update frequency at the back-end. If reactor temperature already announced 100 ms ago and it went to 1051K from 1050K maybe it is better to delay it for a second.