[mumblemumble]

Take a look at that 2nd link in the grandparent post. I remember it as one that was often cited when this debate first went around the Internet half a decade or so ago.

The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed.

One could argue that that is the real antipattern. It'll hurt you big time if you're storing messages in a database table. But it'll hurt you even if you don't. For example, by tracking that kind of information inside of the queue, you're losing the ability to add a second listener without either affecting what messages are being seen by (and therefore the behavior of) the existing listener(s), or modifying the queue itself. Which you don't want to have to do any more than you have to on a critical piece of shared infrastructure like that.

It might be fine if it's an oldschool monolith that's just processing some sort of work queue on multiple threads. But, if you're doing microservices, you're probably trying to keep things more flexible than that, and want to allow them to evolve more independently.

[sbellware]

> The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern.

Indeed.

Lots of good writing on the tradeoffs, for example: https://sookocheff.com/post/messaging/dissecting-sqs-fifo-qu...

At the risk of beating the point to death, the transition from SOA to Microservices is marked by the transition of smart pipes to dumb pipes, and dumb pipes have no knowledge of whether a message has been processed. In the microservices era, the basic presumption is that the transport has no knowledge of the state of the message processors' progress through a queue.

[SideburnsOfDoom]

> the transition from SOA to Microservices is marked by the transition of smart pipes to dumb pipes, and dumb pipes have no knowledge of whether a message has been processed.

That doesn't to seem to be characteristic of Microservices at all. It's more about small bounded context and independent deployment of the service. All of which is completely possible over the "smart pipes" that message queues that I mentioned will give you.

Insisting that this wheel must be re-invented or worked around to make it a microservice seems very odd.

[hmeh]

> That doesn't to seem to be characteristic of Microservices at all.

Many have their own "unique" definition, which is why "microservices" is bordering on no longer meaning anything. Some people say it and mean SOA—the characteristics you mention are pretty much characteristics of SOA when done right, though independent deployment isn't a necessity.

Martin Fowler attempted to codify a definition, and this is inline with the definition that sbellware is referring to. He (and others) refer specifically to smart endpoints and dumb pipes:

https://martinfowler.com/articles/microservices.html#SmartEn...

[sbellware]

Indeed. And having been around in both the SOA period and the Microservices period, and having witnessed the transition, I'm comfortable with maintaining the assertion that "Microservices" as a term was specifically introduced to demarcate the period characterized by big-vendor smart message transports and the period characterized by dumb transports we settled on as a reflection of all that we'd learned by trying to rely on messaging "magic".

It's a very similar transition to the shift from EJB to Hibernate (or EJB to Rails) or from SOAP to REST.

In the mean time, a lot of folks who don't have that background and perspective picked up on Microservices and made a lot of presumptions based on a lot of experience with web apps and web APIs. It's this that made "Microservices" a largely meaningless, muddied term. This is ironic because "Microservices" was introduced as a means to disambiguate the many competing meanings of "Service-Oriented Architecture" that had come into existence as many and vendors wanted to be seen as being involved with it without actually having been involved with it.

So, there's two meanings of "Microservices": One that comes from a background of service architectures and one that comes from a background of web development. My background is in both of them, but I hew to the meaning of "Microservices" which is closer to "SOA without big vendor smart pipes of the mid-to-late 2000s" than "The same old HTTP APIs we see the world through as web developers".

That's largely a lost battle now, just as SOA was lost to the competing interests of message tech vendors.

The term "Autonomous Services" is a much more precise and unambiguous in its intent, and conveys more specifically what's intended by "SOA done right", a.k.a.: "Microservices". And even more specifically, "Evented Autonomous Services" does an even better job of conveying the intent and the implications or the architecture than what "Microservices" can do now.

Or as Adrian Cockroft originally put it: "Loosely-coupled service oriented architecture with bounded contexts".

Knowing what the implications of all those words are, it's quite impossible to look at what is commonly asserted about the meaning of "Microservices" in 2019 and see much left of the great value of what was originally intended. Much of it still remains unlearned.

[SideburnsOfDoom]

> you're putting a key piece of smarts into the queue: Tracking whether a message has been processed.

yes, AWS SQS does that. e.g. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

Are you saying that this is bad? It doesn't seem so.

> you're losing the ability to add a second listener without either affecting what messages are being seen by (and therefore the behavior of) the existing listener(s).But, if you're doing microservices, you're probably trying to keep things more flexible than that, and want to allow them to evolve more independently.

I'm not following. Both AWS SNS/SQS and Azure Event hubs were able to handle that scenario just fine, by design.

In AWS you attach multiple queues to the same SNS endpoint and have a pool of subscribers on each queue, in Azure you declare a "consumer group" and a pool of subscribers in it. Or declare a new consumer group if need be.

We regularly scaled up AWS from 3 listeners to 30 depending on load, and back down again, or replaced instances one by one, and still it guaranteed at-least once delivery (in practice, exactly once in the vast majority of cases) across the subscribers. We _never_ "lost the ability to add another listener" either scaling up in the same pool, or creating a new pool.

[sbellware]

> Are you saying that this is bad? It doesn't seem so.

It's not "bad" per se, but it's not what it seems on the surface.

ACKing a message doesn't mean that the message will not be received more than one time. As long as the smarts for recognizing recycled messages are embedded in the application logic, all will be fine.

Here's a good explanation written about SQS, but applies to all message technologies that work based on ACKs: https://sookocheff.com/post/messaging/dissecting-sqs-fifo-qu...

So, while it's possible to add more consumers, depending on the implementation of the technology's internals for tracking consumer state, you may loose messages or, more typically, receive messages more than once or receive them out of order.

This often happens without the developers' and operators' knowledge. Without foreknowledge of the causes and effects of these things, developers and operators may know there there's some kind of strange intermittent glitch, but don't presume that it's because message processors are processing messages that had already previously been processed (or are not processing messages that had been skipped).

What you can't do is add a new consumer that is interested in processing historical messages beyond the retention window of typical cloud-hosted message transports.

But that's ok because message queues are typically just transports, not message stores, and they serve the needs of moving messages from one place to another. A message store can do that as well, but has different semantics and serves other needs.

So, as long as ACK-based message processors work within the retention window, everything's good. It's when that's not the case that the problems arise. Lost messages and recycled messages are pretty much the only guarantee over the lifetime of a messaging app. As long as that's within tolerances, then it's fine.

Specifically, like all "dumb pipe" technology that came about in the post-SOA age, a message store doesn't use a protocol based on ACKs. Instead, a consumer tracks its own state and does not defer this responsibility to the transport.

And inevitably, if any consumer of any technology wants to guarantee that it never processes a recycled message, this tracking logic has to also be implemented in the application logic of even SQS, RabbitMQ, etc consumers.

There's no such thing as a messaging technology that can guarantee only-once delivery, as you alluded to. There's a good examination of this aspect of messaging and distributed systems here: https://bravenewgeek.com/you-cannot-have-exactly-once-delive...

The best guarantee from message transports that we can hope for amounts to a "maybe". And that's ok. The conditions and countermeasures are well-known. As long as we're not blind-sided and haven't taken "only once" literally, things will be fine.

If we don't realize that application logic is always responsible for making the ultimate decision as to whether to reject a recycled message, we're going to have problems that can be difficult-to-impossible to detect and correct.

Message stores and event stores aren't alternatives to message queues. They're technologies that support architectures that are themselves alternatives to each other.

[SideburnsOfDoom]

> Here's a good explanation written about SQS, but applies to all message technologies that work based on ACKs

Deals with FIFO and ordering. You also mention "historical messages".

Sure, if those are things that you need, then you almost certainly want a different tech to SQS, likely Kafka. I won't disagree with that, except as in making it the focus of the criticism of SQS and similar message queues. It isn't that thing.