"What's a use case where strict ordering is critically important?"
In general, as the use case grows, every use case where the developers did not make explicit and careful provision for ensuring that order is not important, with quite non-trivial effort.
Even a lot of systems whose developers think they have no ordering dependencies are wrong in at least one subtle way without realizing it.
If you need to megascale, you're going to have to bite the bullet and build a system that can handle out-of-order, but there's a lot of systems out there where you don't need megascale, and you can get rid of that "quite non-trivial effort" to deal with out-of-orderness by asking for messages to arrive in order.
To get a sense of just how useful that can be... bear in mind that every time you open a TCP socket instead of a UDP one, you just made exactly that choice, to use an ordered message system when you didn't "need" one. Take a look at everything you do with a TCP socket and think about trying to run it over UDP, and not with something like QUIC that basically adds half of TCP back on it, but with UDP straight-up. That's what kind of things can use in-order delivery... lots of things.
Almost everything can be simplified by guaranteed in-order delivery. It's just that some things can't afford the downsides.
I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped.
I guess all of the systems I build are just built to assume no order/ or to leverage causal ordering, because that feels much easier to reason about - enforcing ordering feels really hard, and like something that a message bus can only do some of the work of.
> I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped.
Think of (almost) any modern protocol built on top of TCP, and you'll see that ordering is critical. (http, smtp, telnet/ssh, etc.)
There's no ordering in HTTP. If you could send a whole HTTP request as a UDP packet you'd get exactly the same protocol (obviously sans WebSockets - but you could work around that).
I'm not sure that ordering doesn't matter for most TCP data. For example, HTTP depends on ordering. Any time you are transmitting messages larger than the size of a packet, you need some degree of ordering, even if it's only to reconstruct the individual messages, when you don't care about the order of the messages.
There are a couple of options without needing guaranteed ordering:
- jobs can have ever increasing ids, workers record the last seen id in one place, and ignore jobs with ids less than last seen
- job results are returned for each job to a supervisor. if a job result doesn't match current expected state, resend job. jobs should be idempotent in case a job is sent multiple times
If the create job is expensive, the latter solution would be less ideal, though.
Creating ever increasing ids reliably at scale is not trivial. You will probably end up having a single server generating these ids which will then become a single point of failure.
That doesn’t necessarily require that the entire queue is totally ordered, but the alternatives (such as Virtual Synchrony) are still considered arcane / research topics.
Now you need to queue up events for some time, reorder them using the timestamp, and then process them. It’s possible, but has overhead in both performance and custom code you’ll have to maintain. If there is no guarantee of order, two separate systems consuming those same events also might get different results, depending on the implementation, that can be problematic
For a single process on one box with one thread you can use something like that.
If you involve more than 1 box that goes out the window. Sometimes you can still get 'one timestamp' by making something else the owner of the timestamp. It also depends on your resolution of time and the process that does the ingesting. For example if that ingest process has more than one thread to handle things you can still get out of order/sametimestamp if not coded correctly.
If events are generated by different processes you cannot really guarantee that time is exactly the same for them, unless you do something fancy to ensure that.
Interesting.
The ordering here is when the event was generated or when the event entered the queue ?
I think the later and so I think the examples here don’t apply without something on top and a trade off
If you have one message source (a single thread or some kind of coordination), and the messages have lower frequency than the timestamp resolution, yes.
The farther you get from that, the more the answer is no.
That won’t work in all cases. For instance, if you get messages from devices which can be reimaged they may have clock skew in a period of time before they’re synchronized again.
But in any case you can't rely on the order of message ingress to your system to represent anything meaningful either? It would have to ensure that the key for defining order would have some hard logical ordering purpose for which time is not relevant or useful.
The order of message ingress can still be meaningful even if device clocks are skew or jump due to rebooting, reimaging, network time sync, frequency drift, etc.
A hard logical order arises from interactions. E.g. if the device receives a message, does something locally, goes through a clock change, and then sends a message dependent on one it fetched earlier, that's a logical order with out-of-order clock.
Or if a device gets a message, processes, sends something to another device, that one processes too then sends another message back to the original source, there's a logical order but with three different clocks. Even if the clocks are synchronised, there will be some drift and the messages may be processed fast enough that the drift puts their timestamps out of order.
That guarantee/assumption can never really be made.
Message A on event a' from system a might be sent to system b effecting event b' and thus message B to be sent by <insert medium here>, and consumed and correlated by consumer software MC on hardware mc.
However system B might take longer to flush it's hardware/software buffer and the message arrives at mc before message A, for example.
I've encountered this many times. That data has no meaning in itself except in the meta.
> Even if the clocks are synchronised, there will be some drift and the messages may be processed fast enough that the drift puts their timestamps out of order.
If you are consuming from sources which you cannot control the accuracy of the clocks, then you must inherently either reduce your reliance on the need for accuracy (many Windows systems have horrendous clock discipline in their software implementation) or find a way of ensuring accuracy. E.g. close proximity NTP or close proximity PTP etc etc.
human conscious thought is local and single-threaded. it takes a lot of experience and training to be able to intuitively reason about non-local multi-threaded computation. if you're smart and humble you can try to simplify the problem by making individual messages independent from each other by e.g. employing redundancy but you still have to be aware that it's even a problem to begin with.
I see it a lot when integrating legacy healthcare systems that effectively operate on state-transition queues with an assumption of in-order processing.
In general, as the use case grows, every use case where the developers did not make explicit and careful provision for ensuring that order is not important, with quite non-trivial effort.
Even a lot of systems whose developers think they have no ordering dependencies are wrong in at least one subtle way without realizing it.
If you need to megascale, you're going to have to bite the bullet and build a system that can handle out-of-order, but there's a lot of systems out there where you don't need megascale, and you can get rid of that "quite non-trivial effort" to deal with out-of-orderness by asking for messages to arrive in order.
To get a sense of just how useful that can be... bear in mind that every time you open a TCP socket instead of a UDP one, you just made exactly that choice, to use an ordered message system when you didn't "need" one. Take a look at everything you do with a TCP socket and think about trying to run it over UDP, and not with something like QUIC that basically adds half of TCP back on it, but with UDP straight-up. That's what kind of things can use in-order delivery... lots of things.
Almost everything can be simplified by guaranteed in-order delivery. It's just that some things can't afford the downsides.