[paulgb]

NATS is great, we use it in another project, Plane (https://plane.dev). The reasons I didn’t use it instead of making DriftDB:

- In NATS, unless you set up authentication, any user can subscribe to “>” and get a firehose of every message, even if they don’t know the room IDs.

- NATS Jetstream supports rollups, but they roll up the entire stream, rather than up to a certain sequence number. This would break our ability to do leaderless compaction.

[bruth]

Was not aware of Plane, nice! Regarding the two points:

- A unique room ID/subject is a form of authentication. Essentially anyone having that unique identifier can join, akin to a token. This is straightforward to setup in NATS avoiding the ">" for all problem (which I may now need to write a blog post about ;-)

- Rollups are supported on a per-subject basis. Each room could be modeled as a subject and individually rolled up.

[paulgb]

Re. #1, can you elaborate? The tracking issue for this is still open[1]. As far as I can tell this is a hard blocker for any use case where a random user on the web can connect to NATS, since it means that user can wiretap any room without knowing the room ID.

Re #2, the problem is that a rollup of a subject in NATS rolls up the whole subject, so there’s a race condition if you try to use it the way DriftDB uses it. If one client is computing a compaction while another client sends a message, that message will be erased by the compaction.

This works if a single producer is writing to a stream, because that producer can stop emitting messages during the compaction. But in our case, each client can produce messages at any time.

DriftDB solves this by sending a sequence number alongside the rollup of the last message included in the rollup, and the server preserves messages after that sequence number.

[1] https://github.com/nats-io/nats-server/issues/2667

[bruth]

#1: This isn't a hard requirement to achieve the desired permissions. For the DriftDB use case, my understanding is that all members in the room have full pub/sub k/v permissions, so that could be achieved by declaring a new permission pinned to the room when the room is created or joined (this can be done dynamically without a config file reload).

#2 Publishes do support optimistic concurrency control using the `Nats-Last-Expected-Sequence` (stream level) or `Nats-Last-Expected-Subject-Sequence` for the subject-level. This ensures to concurrent publishes will be serialized and all but one is rejected with "conflict wrong sequence" error. For example headers in Rust[0] and WS[1]

[0]: https://docs.rs/async-nats/latest/async_nats/header/index.ht...

[paulgb]

#1 your understanding of the DriftDB permission model is correct, but I’m not sure what declaring a new permission at runtime entails? Would I be creating a new bearer token for each room, and attaching the room’s permissions to it?

#2 the optimistic concurrency headers don’t solve the problem here, e.g.:

- I increment a counter (seq: 1)

- I increment a counter again (seq: 2, expected last sequence: 1)

- I begin computing a snapshot, resulting in a counter value of 2

- You increment the counter (seq: 3, expected last sequence: 2)

- I complete the snapshot and publish it with a Nats-Rollup header

- Your event has been lost, the counter value is now 2

[bruth]

#1: Correct, I putting together an example this week to show what this looks like. Pretty straightforward.

#2: You can combine rollup and expected last sequence header to prevent this, unless I am missing another subtle detail?

(I am enjoying this thread FWIW :)

[paulgb]

Cool, looking forward to the example :)

I actually didn’t realize that the expected last sequence could be combined with nats-rollup. In the example above (as I understand it) if we added that to the snapshot, NATS would throw out the snapshot, so the log would still be correct, but the work of snapshotting it and sending it over the wire would be lost. If messages were frequent enough, and/or snapshots took a while to compute/transfer, you might never have a roll-up succeed.

Our approach is that a roll-up will always succeed, we just preserve any messages in the stream with a sequence number greater than the one provided.

(I’m enjoying it too, and am a user of NATS so I’m happy to learn things I didn’t know about it :)