[shados]

If you do it at a small scale, where a team of a handful engineers is responsible for following/maintaining/debugging countless distinct systems that are only linked through Kafka, its insane.

if you have hundreds or thousands of engineers in more teams than you can count, and each team is responsible for a few event producers or consumers and have the time and resources to dedicate to be experts about their own systems and their interfaces/boundaries, it's an amazing way to scale your organization. It really does work.

[kingdomcome50]

I understand what you are saying here. But... I am having trouble determining the implication.

It seems like you are saying that an EDA is only useful when no one has to understand the system in its entirety? That can't possibly be useful can it?

The implication is that an EDA is inappropriate if I have a system with 1000 services and 5 engineers, but amazing if I have 500 engineers? Isn't the purpose of a technical architecture to optimize for the former (assuming engineering is a cost-center)?

The former (small-scale) version you describe is really just an optimization for the latter (large-scale) version you describe a la "We have simplified our systems so much that we no longer need as many engineers to maintain it".

In a way, your comment serves to substantiate the point of its parent in that it asserts that choosing an EDA requires more engineers per service.

I must be misunderstanding your point.

[bcrosby95]

> Isn't the purpose of a technical architecture to optimize for the former (assuming engineering is a cost-center)?

The companies that developed these things are employing hundreds or thousands of engineers. So, yeah, they are optimizing for engineering costs when you have thousands of engineers. They are building systems that cannot be fully understood by a single person. If you don't have thousands of engineers, their solutions to their problem won't necessarily be your solution to your problem.

Complex beasts like microservices and event driven architecture are about enabling huge engineering departments to make changes to huge systems without constantly stepping on eachothers toes.

Ignoring headcount, they are actually very inefficient because systems designed like this necessarily has tons of duplicated effort across the organization. Those inefficiencies are made up by reducing the amount of communication required between thousands of people and hundreds of teams to change and maintain the system. But if your department isn't huge then the communication overhead is less than the duplicated effort, which tends to make these designs a poor choice.

These sorts of things usually do not scale down. If you have 1,000 services and 5 engineers there's a 99.999% chance you're doing it wrong. 5 engineers is not even enough employees to manage, monitor, and build the requisite tooling to make efficient use of a system distributed across 1,000 services, much less build and understand the system as a whole.

[shados]

No, if you can have 1000 services maintained by 5 eng, you're doing fine.

The problem organizations hit is such: you have 2-3 huge services maintained by 5 engineers. The company grows, the services grow. 5 eng become 10 eng become 100. It's still the same 2 or 3 services but much, much bigger. You get to a point where people are getting in each other's way, and every additional person you had provided less value than the people before them. They have to work within the boundaries of the system and coordinate with other people.

It's all in the same programming language so you are limited by one pool of potential hires. Tech debt affects everyone. Deploys affect everyone. Downtime affect everyone.

So you split it to solve some of these. Then split it again. And again. And again. You keep splitting until someone can reasonably wrap their head around a single atomic piece, from implementation to deployment. The piece can be rewritten from scratch at any time without impacting any other piece, in any language, on any infrastructure, with any framework, to optimize for the people working on it.

That's great, but that piece still has to be part of the whole. These event buses are an elegant way to keep them together without adding complexity at the individual piece level.

Yeah, it means people might not the understand the whole system anymore, but do they need to? It's like people organization. The CEO doesn't know the intricacies of the HR system. They know it at a macro level but defer to the HR manager for the details. Same thing here, but with software.

[bob1029]

Can confirm that message buses aren't just buzzwords and are literally the only feasible way to combine business systems in the most complex of enterprises. I have seen a messaging domain with over 1000 different event types to deal with. Gigabytes of messages per hour. 24/7/365. Tens of thousands of different system interfaces all talking together using synchronous RPC semantics without experiencing any errors for days at a time. Systems spread across the planet. It's the perfect solution if you have the resources to maintain it.