[bruth]

Nice, have you come across NATS? https://nats.io. The server natively supports WebSockets. There are many clients including Deno, Node, WebSockets, Rust, Go, C, Python, etc.

In addition to stateless messaging, it supports durable streams, and optimized API layers on top like key-value, and object storage.

The server also natively supports MQTT 3.1.1.

[SpaghettiX]

Nats is not something I see as a competitor for external clients (browsers, mobile apps), primarily because it doesn't handle reconnections / message delivery / quality-of-service / at-least-once or exactly-once delivery (except for MQTT).

> When the connection is lost, your application would have to re-create it and all subscriptions if any. https://github.com/nats-io/stan.go#connection-status

Therefore, I don't see what it adds here. It seems designed for service communication, not client-server. They also don't list browsers as a use case https://docs.nats.io/nats-concepts/overview#use-cases. (though it is of course possible, it's just not ideal IMHO.)

They still have a js/browser client library though if you want to use them: https://github.com/nats-io/nats.ws. And yes, their servers "have websocket support".

[bruth]

In fact it does all of these things now properly! STAN (NATS Streaming) was deprecated two years ago in favor of a new embedded subsystem called JetStream: https://docs.nats.io/nats-concepts/jetstream released in March 2021.

[SpaghettiX]

Even with NATS jetstream, NATS has a focus on service communication.

"It supports websockets" and "qos" does not mean it will work robustly with web apps if nobody uses NATS for that use case. See https://github.com/nats-io/nats.ws/issues/172 for an example issue. If NATS is not used for websockets in browsers, it will have a mine field of issues to fix. And what about all the other clients (mobile, mobile web)? Sure there may be a NATS client library for it, but it won't handle user connectivity issues, because again it's aimed at service communication where the network is great.

People are using NATS in kubernetes, not web browsers.

[bruth]

> Even with NATS jetstream, NATS has a focus on service communication.

It indeed excels at service communication as well. However, a core use case for NATS is the edge, be it your definition (browsers and mobile), but also in cars, factories, tractors, low-orbit satellites, etc, whether it is running on Kubernetes, k3s, or bare metal.

The issue you called out is a Firefox-specific issue, but it will be addressed and not indicative of an inherit limitation of NATS.

Check out this playlist of a live event I organized last fall with a variety of live demos: https://youtube.com/playlist?list=PLgqCaaYodvKY6xRbvB6ffON0_...

[SpaghettiX]

Currently, the only people I see talking about NATS for edge is Synadia - they're also not very specific. In theory/documentation, "edge" is a core NATS use case, but in practice why does NATS compare themselves to Kafka and microservices? Most of that playlist is not edge-focused. Can you explain what concepts and problems you have to solve to support the "edge" - none of that is in your website.

> The issue you called out is a Firefox-specific issue, but it will be addressed and not indicative of an inherit limitation of NATS.

My point is NATS is not being used in browsers, mobile apps or edge use cases. It doesn't even explore the concepts. It looks like it doesn't care about Firefox. For IoT, what does NATS bring on top of MQTT? NATS ends up being an MQTT broker so it will have to compete with all of them.

Why don't you start comparing yourself to products and technology that serve the edge (other edge-focused companies (ably, pusher, pubnub), and other MQTT brokers)?

Side note: Would appreciate it if you disclosed your affiliation with Synadia and NATS before advertising it.

[bruth]

> Would appreciate it if you disclosed your affiliation with Synadia and NATS before advertising it.

Fair point, but I did not mention Synadia. FWIW, I have been a NATS user for seven+ years prior to joining Synadia so I was speaking on behalf of myself and experience with the tech.

Also fair point that the nats.io website does not highlight this strongly. The NATS maintainers are aware (nearly all employed by Synadia) and we are working on it.

I disagree that simply because it is not advertised as a "edge" technology that it is not one of the best-in-class techs for edge. It simply means, we are doing a poor job at awareness.

> Why don't you start comparing yourself to products and technology that serve the edge

The vast majority of people and customers compare NATS to Kafka and the variety of variants out there. Once the push on edge occurs, I suspect comparison to these other tech will occur.

To be clear, I am not looking for a "winner" in this discussion, rather my original comment was to correct a gap in understanding of what NATS is capable of.

[autobeam]

actually, NATS can be used in the browser in a way that eliminates the need to REST and offers both client/server communication as well as realtime functionality, a big benefit of that that full stack developers, will be able to communicate between web app and backend services in the same way that services talk to each other in a secure and reliable way, i wrote a blog post about that here: https://www.ahmed.wiki/blog/nats-more-productivity-client-de...

Not saying that driftDB is not cool, it is a nice tool, the point is that NATS has a great way of streamlining the communication between all components of a system including client apps, (react + flutter) in my case

[SpaghettiX]

> a big benefit of that that full stack developers, will be able to communicate between web app and backend services in the same way that services talk to each other in a secure and reliable way

In practice, I have not seen that manifest as a benefit. Services have a dramatically different environment than edge devices. Tools built for services (NATS, Kafka, gRPC) do not translate well to the edge. The latter is used by a group of people who don't care or understand the edge-edge-cases: when a user drives through a tunnel and is disconnected, or when they restarts their device, or is throttled by an OS, etc. One issue I found with grpc-web (the alternative to grpc that supports browsers) is that it's severely limited by connection count by the browser - making streams completely useless). Also, grpc-web is neglected by Google.

NATS does not look ready, and is not designed for use, in browsers or mobile apps. It's not a use case that Synadia/NATS care much enough to even mention on their website.

> (react + flutter) in my case

How are you using NATS in Flutter? Using the client library that hasn't been updated in 20 months, with no link the repository and 4 upvotes? Or writing your own custom library to connect using websockets. If you use websockets directly, you'll be writing extra code to handle disconnections, retries, qos, etc.

[paulgb]

How do you prevent a malicious user from taking the wss endpoint you’re using, subscribing to a wildcard, and seeing messages intended for other clients?

[SpaghettiX]

They also don't list themselves as competitors to products that do target this market: ably, firebase messaging, pubnub, pusher. Instead, they compare themselves to Kafka, RabbitMQ, Pulsar, and gRPC, none of which work well on browsers. (Yes grpc-web "works" on the browser, but I suggest everyone avoids it.)

https://docs.nats.io/nats-concepts/overview/compare-nats

[paulgb]

NATS is great, we use it in another project, Plane (https://plane.dev). The reasons I didn’t use it instead of making DriftDB:

- In NATS, unless you set up authentication, any user can subscribe to “>” and get a firehose of every message, even if they don’t know the room IDs.

- NATS Jetstream supports rollups, but they roll up the entire stream, rather than up to a certain sequence number. This would break our ability to do leaderless compaction.

[bruth]

Was not aware of Plane, nice! Regarding the two points:

- A unique room ID/subject is a form of authentication. Essentially anyone having that unique identifier can join, akin to a token. This is straightforward to setup in NATS avoiding the ">" for all problem (which I may now need to write a blog post about ;-)

- Rollups are supported on a per-subject basis. Each room could be modeled as a subject and individually rolled up.

[paulgb]

Re. #1, can you elaborate? The tracking issue for this is still open[1]. As far as I can tell this is a hard blocker for any use case where a random user on the web can connect to NATS, since it means that user can wiretap any room without knowing the room ID.

Re #2, the problem is that a rollup of a subject in NATS rolls up the whole subject, so there’s a race condition if you try to use it the way DriftDB uses it. If one client is computing a compaction while another client sends a message, that message will be erased by the compaction.

This works if a single producer is writing to a stream, because that producer can stop emitting messages during the compaction. But in our case, each client can produce messages at any time.

DriftDB solves this by sending a sequence number alongside the rollup of the last message included in the rollup, and the server preserves messages after that sequence number.

[1] https://github.com/nats-io/nats-server/issues/2667

[bruth]

#1: This isn't a hard requirement to achieve the desired permissions. For the DriftDB use case, my understanding is that all members in the room have full pub/sub k/v permissions, so that could be achieved by declaring a new permission pinned to the room when the room is created or joined (this can be done dynamically without a config file reload).

#2 Publishes do support optimistic concurrency control using the `Nats-Last-Expected-Sequence` (stream level) or `Nats-Last-Expected-Subject-Sequence` for the subject-level. This ensures to concurrent publishes will be serialized and all but one is rejected with "conflict wrong sequence" error. For example headers in Rust[0] and WS[1]

[0]: https://docs.rs/async-nats/latest/async_nats/header/index.ht...

[paulgb]

#1 your understanding of the DriftDB permission model is correct, but I’m not sure what declaring a new permission at runtime entails? Would I be creating a new bearer token for each room, and attaching the room’s permissions to it?

#2 the optimistic concurrency headers don’t solve the problem here, e.g.:

- I increment a counter (seq: 1)

- I increment a counter again (seq: 2, expected last sequence: 1)

- I begin computing a snapshot, resulting in a counter value of 2

- You increment the counter (seq: 3, expected last sequence: 2)

- I complete the snapshot and publish it with a Nats-Rollup header

- Your event has been lost, the counter value is now 2

[bruth]

#1: Correct, I putting together an example this week to show what this looks like. Pretty straightforward.

#2: You can combine rollup and expected last sequence header to prevent this, unless I am missing another subtle detail?

(I am enjoying this thread FWIW :)