[DoneWithAllThat]

I’ve worked for decades now as both a developer and an SRE and have never once thought either “man I wish these microservices were monolithic” nor “man this monolith is so great I’m glad it’s not a collection of microservices”. These kinds of articles seem to be written for people who work in environments I’ve never even heard of let alone experienced.

[ericbarrett]

I've seen the degenerate states of both extremes:

- a monolith with too many engineers contributing: no continuous deployment but rather "release day," which was a shitshow; said day was an extensive process where hundreds of engineers (anybody who was on git blame) had to be online and sitting by for four hours while it rolled; the release team often had to hand-revert bad patches; bullying of engineers who "broke the build" reached levels that would raise HR eyebrows; there were still often rollbacks and site breakages.

- microservice hell: there were often 2-3 APIs for the same service; platform engineering (for the protobuf RPC generation) had to support five different languages; security auditing was NP-hard; every team had its own release process; services that were still highly used were "deprecated" and left languishing for years until somebody took up the mantle and released their own parallel service that did nearly the same thing but with a different API, so now other services had to use both; etc.

"But that wouldn't happen at $my_company, we know the pitfalls and we'd never be that bad at engineering!" Sure...sure. That's what these companies said to themselves too :)

[otagekki]

I have also known both albeit on a lesser scale.

At a company we as a dev team had to maintain a constellation of desktop applications and homemade ETL Airflow-like software that scaled really badly. For the time I'd been there I had managed to automate deployment almost totally on my own. Sadly it had been somewhat controversial as it was done at the expense of other development tasks (my manager agreed but not his nontechnical manager), and probably too late too. Before I had had enough grasp on the complicated context we had to deliver on a big project and needless to say it was a shitshow and we developers took basically all the blame, with pretty much nothing to say back.

The tech stack and code had been in a state of neglect for 8+ years but that was not an acceptable answer at that time, it seemed, and I'd only been there for 6 months when it happened.

On the other end of the spectrum, at the above's parent company, I had to maintain several major versions of an API on several cloud regions. Undocumented peculiarities regarding the object storage that was specific to one region caused a production incident that took the dev and support team 3 days to untangle and solve.

I have similar stories on monorepo vs. multirepo too

[DJBunnies]

I think that all the time.

Or more specifically, "I wish these services were a monolith so we could have better type checks/easier logging/debugging."

[bluefirebrand]

"I wish these services were a monolith so there was any amount of discoverability at all, and I don't have to beg for time from busy people on other teams to help me use their undocumented APIs"

[arinlen]

> I wish these services were a monolith so there was any amount of discoverability at all,

Why do you conflate microservices with discoverability? What's wrong with simply calling a web service?

> (...) and I don't have to beg for time from busy people on other teams to help me use their undocumented APIs

And you believe that the same hypothetical team you claim doesn't document their API would all of a sudden documented all its internal?

[bluefirebrand]

If it was a monolith I would have access to the same codebase, so I could just go look at stuff in the code?

Versus microservices where I might not have permissions to look at their repos.

[thaneross]

It's a fascinating set of priories that lead to these policies.

"We trust you enough to run your code in production but we can't let you read another team's code because you might steal it. Yes, the odds that anyone wants to steal the code for this mundane microservice are staggeringly low... and yes you're going to be less productive and more prone to serious bugs by being in the dark, but that's your problem."

[bluefirebrand]

I honestly don't even know if it's "you might steal it"

It's just this vague notion of "if you aren't contributing to this repo you don't need to access it" or something.

[KronisLV]

When I'm forced to work with a monolith that has 4000+ source files and takes a few minutes to even start up, and has about 20 configuration files that all need to be setup properly for it to even start, I wish for microservices.

When I'm forced to work with microservices that need Skaffold and Helm charts just to run locally, but with the configuration in the monorepo being kind of mismanaged and strewn around a bunch of folders with no documentation, in addition to debugging in the IDE not working because nobody set it all up, I wish for monoliths.

Really, you can have good monolithic systems and you can have good microservices as well, in addition to something in the middle: https://blog.kronis.dev/articles/modulith-because-we-need-to... (actually the first blog post that I wrote, the casual language probably shows its age).

But there can also be plenty of poorly developed projects with either. It just so happens that people hate monoliths more in the mainstream culture because most of the older and worse projects out there were monolithic, much like many hate Java and other languages because of being exposed to bad legacy projects written in them: https://earthly.dev/blog/brown-green-language/

Just wait for 5-10 years and people's disposition towards both monoliths and microservices will even out, the advantages and disadvantages of either will become clearly recognized and the hype will shift towards something like serverless. Much like now we know the advantages and disadvantages of languages with/without GC as well as higher/lower abtraction levels pretty well (consider Python vs Rust, for example).

Maybe things will slow down a bit because Kubernetes will also become a bit easier to use, possibly thanks to projects like K3s, K0s, Portainer and Rancher.

[barking_biscuit]

It turns out the secret is just not writing crappy software and having crappy processes. It turns out the caveat is that the conditional probability of those two things is vanishingly small, so regardless of monolith or microservices, most devs are exposed to crappy stuff and because their sample sizes are tiny they assume well we just did it wrong, when in fact they did it average and average is crappy for any and all architectures.

[KronisLV]

> ...when in fact they did it average and average is crappy for any and all architectures.

I feel like this is probably more true than many of the people around would like to admit.

Very much like how there are plenty of proponents of DRY and SOLID but plenty of bad systems still get built regardless, even when trying to follow any number of principles along the way.

So you'd have to decouple what an architecture is supposed to (realistically) be, from any company-specific interpretation of that, which may not actually be doable. It's not so much the "No true Scotsman" fallacy (which comes to mind), but rather the fact that things almost always differ in theory in practice.

As an unrelated example, it would also be really useful to generate implementations from code models, like OpenAPI spec --> usable API client in language/framework X, or OpenAPI spec --> usable service stubs for the server in language/framework X, but in practice it never works out and most model driven approaches remain on paper.

Similarly, there are lots of ways to screw up your application architecture regardless of the specifics. Talking about which type is easier to screw up and in what ways, however would be a useful conversation to have!

[_hcuq]

I wonder if part of the problem is the always new technologies used. Maybe if the world had stuck with some sort of improved version of C and got really good at it, with all kinds of great tools and techniques, we might be able to deliver code efficiently.

[barking_biscuit]

I definitely think that's an aspect of it, but it's not one we can really do a whole lot about. The nature of what we do and the skillsets we have mean when we encounter problems we then come up with solutions and those solutions have their own problems, so it just infinitely recurses.

I think that's a lot more controllable on an individual level than on an industry level. I'm sort of attempting to do that myself in some ways by just picking a stack and working my way through building something in it end to end covering absolutely every aspect I can imagine in terms of software engineering "jobs to be done" and as I build it, going through industry go-to resources of how to best do X and properly implementing the advice. I think if more of us bothered to do that we'd be far more capable of starting software companies off at a much more advanced stage of technical maturity and hence with less problems that arise from the hacky shit we cobble together under financial pressure to ship, ship, ship. That would ideally result in significantly less churn.

Essentially, yes. Just take something and get ultra good at it and then stick with that thing.

[jnash]

Nothing stops microservices from having the same problems. Plus all the extra problems you get with distributed systems (timing issues, logical dependencies etc.)

[throwaway894345]

Same. I keep hearing that people have had bad experiences with microservices, but I'm not sure what those bad experiences are. I certainly don't relate to them. We had a monolith, and it devolved into a mess--managers would insist on people taking dependencies on stuff they didn't own in the name of expedience ("yeah, we'll totally circle back and do it the right way lol"). Microservices kept things neat by making dumb things hard. We were also able to deploy individual services much more frequently without needing to coordinate with a bunch of other teams, and builds, tests, etc were much faster helping shorten our iteration loops.

Local development environments were a bit more tedious in certain cases, but that was the only issue I recall.

[willcipriano]

Microservices suck when you need to make a process faster, so you run a profiler and figure out that 90% of the workload is handling the http requests between the services.

[bb88]

I feel like HTTP is too heavyweight for microservices. For monoliths, it's great. And the only way to proceed in your case is to convert microservices into mini-monoliths. Particularly the authentication side, which needs to be revalidated with every request.

gRPC might be better, but supposedly so was SOAP, CORBA, DCOM, XMLRPC, and finally REST.

[throwaway894345]

How does gRPC make authentication lighter-weight? Presumably you still have a token in either case which needs to be validated? There aren't many places where parsing HTTP requests is really a significant bottleneck, and I'd guess the majority of the remainder are an architectural problem (someone using a "get" endpoint in a tight loop instead of a "list" endpoint or some such).

[bb88]

Technically you could auth the connection using say, a custom SSL cert, and the connection isn't made unless the SSL cert is recognized.

There are other ways to do it as well where the socket connection is only authed once.

[anonymoushn]

At a multibillion dollar company, local development is impossible, staging doesn't exist, most stack traces from production are truncated because the log aggregator cannot handle log entries as large as java stack traces, and most of the work involves migrating from talking to the database to talking to a microservice that exposes a single table of the database. What are joins?

At another company that was acquired by Amazon, the "user service" team (so again this is a microservice that exposes a single table of a database, this time with a two-pizza team dedicated entirely to that microservice) told us that we couldn't just query the user service when we wanted to render a page containing a username given a user id, because that was too many queries. Product demands from a VP dictated that we didn't have time to set up our own caching layer for their service (is this the responsibility of every team other than theirs?), so we shipped the feature with the usernames saved in our own DB, and now when users change their usernames the old name will appear in the pages for our feature, depending on when the pages were created.

[barking_biscuit]

Obligatory repost: https://www.youtube.com/watch?v=y8OnoxKotPQ&ab_channel=KRAZA...

[deterministic]

If you don’t have the skills to implement a well structured and easy to maintain monolith then you most definitely don’t have the skills to implement a distributed more complex implementation of the same system.

[throwaway894345]

On the contrary. An experienced architect defines the boundaries and then the microservice architecture enforced them in a way that a monolithic architecture cannot.

[deterministic]

On the contrary. Using libraries with clearly defined interfaces makes it easier to combine those libraries into a single executable, reducing deployment and debugging complexity. It also dramatically improves performance given the lack of intermediate encoding/decoding protocol code and TCP/UDP hops. Experienced seasoned Architects knows this and work hard to remove unnecessary distributed complexity from systems they design. While “beginner experts” always pick the most complex “solution” for the job at hand. The worst spaghetti code I have ever seen in my 30+ year career is a 20+ years old Microservices architecture. So I can’t wait to read the future “We had a problem so we chose micro-services. Now we have N distributed problems + K impossible to debug obscure timing issues” articles.

[barking_biscuit]

My 2c is that an experience architect would know that it would be ideal if they could do this and be right, but in practice they aren't likely to get those boundaries correct up-front and it takes time and experience working in the domain to see what those boundaries truly ought to be. Also, it's perfectly possible for monolithic applications to enforce boundaries as strict as microservices do via a "modular monolith" approach where the domain contracts are marked as public, but all the domain logic is marked as internal, and a framework of sorts enforces that each module only has access to either it's own schema in a shared database, or it's own separate database.

These days I think that's actually where everything should start off. With a good implementation of this you can transition relatively painlessly to a distributed system, and in the initial phases the simplicity of the build/test/deploy cycle massively lowers the overhead compared to a microservices approach while the strong isolation of domain logic gives you the ability to reason about changes to your domain not affecting other areas of the system.

Having worked in both small companies with terrible monoliths, small companies with terrible microservices, big companies with terrible monoliths and big companies with semi-decent microservices, and then implementing a modular monolith myself, I've seen enough to know that it's a solid, solid starting point. Most companies are pretty small, and the difference between an engineering org of 40 engineers trying microservices vs an engineering org of 4000 engineers doing microservices is on such a different level that people who haven't experienced both likely can't appreciate the data point that they're missing.

I love the strong isolation that microservices provide, and they can be super beneficial, but done at the wrong scale, the wrong level of technical maturity, or the wrong time and they're a huge drag. By the same token a monolith that grows too large for too long is equally a drag, but it's one that's exponentially more tolerable in an engineering org of 40 engineers than it is an engineering org of 4000 engineers. The key thing isn't which architecture you pick, it's the ability to be able to transition and adapt at the right point in time. Modular monoliths buy you this opportunity at a fraction of the price and for much less risk.

[jnash]

100% agree. You can write bad code using both Monoliths and Microservices. However Microservices are inherently more complex and slower than Monoliths (because of the added encoding/decoding and network overhead).

[throwaway894345]

This is probably true if you have good engineering discipline in the monolith case such that your team adheres to good architecture. In the microservices case, as long as your architect is competent, you can draw sane lines between components and it's dramatically harder for IDGAF managers and developers to violate them. Everywhere I've worked, the microservices approach ends up being considerably simpler precisely because it makes it difficult to do things that are probably bad ideas in the first place (it's the same concept with static typing, by the way--a sufficiently smart team may not need explicit types, but in practice most teams aren't sufficiently smart, and the types function as rails to keep people from doing dumb things). Every place I've worked, the network and complexity overheads are paid for many times over by the simplicity and performance of preventing people from doing dumb things.

It's also worth noting that microservices are more secure and reliable as well, since there's no vulnerable component that has access to all of the secrets nor a bunch of components that ought to be non-critical but actually are critical because anything can bring down the entire application for all other components (e.g., someone makes a sync call in some leaf component that breaks the event loop for the whole application, with failures cascading across your replicas).

[barking_biscuit]

I think the critical aspect of whether or not you need them and hence whether or not they represent a good set of trade-offs, is on what dimensions are you optimizing for.

My previous employer was a small engineering org of around 40 engineers with a monolith of below average quality, trying to go in a microservices direction and not really having that pan out a great deal due to being unable to spare the engineering bandwidth to invest enough in managing the complexity while still having to deal with the monolith.

My current employer is an engineering org of around 4000 engineers that one two headed monolith that makes up the original product (web app + api share some code and are monoliths in their own right) and also very successfully uses microservices for everything else and there is a lot of them.

At my previous employer dealing with the monolith was fine even though the code was rather horrid, and we would release twice a week. The "microservices" I wound up building/working on there were not worth the overhead / complexity added at all and we would have been far, far, far better off investing in cleaning up the monolith to make it safer/easier to work in.

At my current employer dealing with the monolith is not fine. You have to book releases in a calendar, it's scary, it's giant, and despite multiple releases of it per day, there are so many other teams that book releases for it that you have to queue up to get your turn and it's done that way for safety. Yes, we also use feature flagging too. It's also vastly more difficult to know/find who you have to communicate with about changes to the monoliths because unlike a tiny engineering org, that information simply doesn't fit in your head. Here, the microservices we own are awesome in comparison. They are small code bases that are easy to reason about, our organization invested a massive amount in the technical maturity needed to operate, maintain, and be able to handle the complexity, and we have continuous deployment pipelines that mean we can safely release multiple times per day and when PRs are merged they are shipped all the way through to production in around 15 minutes.

So, when you say "more complex and slower" you have to ask yourself "than what?" There is no "one is better than the other", there is only "every tool is the right tool for some job" and "every tool is also the wrong tool for almost any other job it was not designed for" and your job is to understand exactly what tool your specific situation actually calls for.

This is why I like modular monoliths as a starting point, as they're essentially a multi-tool. A half-way compromise which is not as easy to use as a full-blown dedicated tool, but is versatile enough that if it's the only tool you have and you have no idea what job you're going to need to do, then you've probably made an adequate choice regardless of what the situation calls for in the end. And if you outgrow your multi-tool, you can much more easily swap it out for a set of it's dedicated equivalents.

[arinlen]

> Same. I keep hearing that people have had bad experiences with microservices, but I'm not sure what those bad experiences are. I certainly don't relate to them.

Likewise. When I see people complaining about microservices, more often what I see is actually poorly thought-through strawmen aimed at distributed systems, which boil down to "having to do network requests is bad".

I wonder why attacking the "microservices" buzzword gets these people on rage mode but the sight of a web app calling a dozen APIs somehow doesn't make them bat an eye.

[jnash]

That sounds like a "True Scotsman" argument. The fact is that microservices are inherently and objectively more complex than Monoliths. So if you can't build a well engineered Monolith then you are pretty much guaranteed to build an even worse Microservices system (because you now have the added encoding/decoding, network hops, latency and timing issues, no guaranteed data consistency, no global commit guarantees etc. etc. etc.)

[throwaway894345]

I can believe that monoliths have a lower theoretical complexity floor than microservice architectures, but I don't think any shops are coming close to that theoretical complexity floor irrespective of their architecture, so we need to think about the actual, practical complexity of applications in each architecture.

Specifically, I think the strong boundaries between components better facilitates maintainability--as with dynamic typing, the absence of rails allows people to more easily make bad decisions. In the case of monoliths, this usually looks like an inexperienced developer (especially at the bequest of an unscrupulous manager who swears that we will definitely not do this again and will fix in a subsequent release) taking a dependency on another system's private internals. In other words, I assert that given a good architect and an average team, a microservice architecture will be less complex than a monolith despite marshaling data over a network. In order to have a well-engineered monolith, you need good architects and a good team (no unscrupulous managers who are empowered to compromise the architecture in the name of expedience).

> no guaranteed data consistency, no global commit guarantees

You don't get these for free with monoliths either. I've seen a lot of monoliths choc full of race conditions which don't manifest during development (small contrived test cases running on unburdened systems). Paradoxically, because microservices have to marshal data over a network, these consistency issues are more likely to appear. Moreover, people who work on microservices are more likely to think about race conditions precisely because the system is distributed (indeed, the architects think about this stuff when they lay out the interfaces).

[mbrodersen]

A friend of mine is working on a 20 years old Microservices application. It is literally the worst piece of garbage system I have ever heard about. I am sure my friend would have loved for it to be a monolith instead of 50+ services with massively complicated distributed dependencies and timing issues.

[nailer]

Really? You’ve never seen an over-engineered microservices design that could be a much simpler express app?

[fud101]

What is an express app?

[nailer]

An app in express.js, a web framework.

[honkycat]

There are a lot of small shops who aren't dealing with scale, and assume that what works for them works for everyone.

Personally, I think micro-services should be approached very carefully but I understand the idea of them.

[jnash]

Companies like Jane Street use Monoliths to handle millions of transactions per second. How many transactions per second do you have?

[atleta]

It still depends on the organization size (or the size of the user base, or the complexity of the product). So you may only have experience with services where it was the right choice.

I did see two instances where it wasn't. (I mainly work with/for small companies, startups.) In one instance I was called in as a tech lead/expert for a small startup having a kind of a product/software crisis. They've been working on their service for a year or too (yes, way too long) and 2-3 months before the planned release at some random conference (Slush, TNW or whatnot) one of the developers figured out that the whole codebase was a piece of shit and there was no way they could be ready in time, but they should rewrite it as a collection of node microservices and that could work. The monolith they had was PHP, just to add to the fun, so switching over would mean switching languages too.

The guy, he was a smart and motivated chap, even started implementing one service in his free time, the user registration/user handling I think (the least important and least complex one, of course) which somehow screwed up the monotlith and made it start crashing. (Or so they said, I don't know what was up with that.)

Obviously, it was a 100% stupid idea and we went on with fixing their development process (starting to do scrum and teaching the stakeholders that they need to stop phoning the developers directly and asking for features, fixes, introducing automated testing, etc.) Oh, and it was a team of 2-2.5 people. (With the group manager doing some backend work too, but also managing another, totally unrelated project for another client.)

The other one was a bit different story, where I just shared my insights over a call. A guy I've known took over a project that was built by a small team (2-5 people, can't remember) for a startup and wanted some external opinion for himself and the founder. That one was built as a set of microservices but they did have all kinds of stability issues. The idea was that it had to be very-very-super scalable. Because, you know you launch and they will come and there's nothing worse than not being able to handle the load. Except, there is: they had been building the thing for over 2 years back then.

It was an online medical consultation solution (you describe your problem, pick a doctor, do a f2f call and pay by the time). The funny thing is that I've built a very similar system, as a startup cofounder, 3-4 years earlier for psychology consultation with the help of 2 other guys, who didn't even work full time (one of them came after the first one quit). The MVP was up in, I think, 2 maybe 3 months. Ours was a monolith-ish thing and theirs definitely looked better, maybe scaled better and would have been cheaper to operate at scale (we used an external service for the video calls). But ours was a lot cheaper to build and launch and we could test (validate) our solution a lot earlier with real customers.

If it worked out, we could have started breaking it down into multiple services as/if/when needed.

[arinlen]

> (...) The monolith they had was PHP, just to add to the fun, so switching over would mean switching languages too.

Why do you feel this is relevant, let alone detrimental to the idea of microservices? It looks to me that it's one of the primary positive traits.

> The guy, he was a smart and motivated chap, even started implementing one service in his free time (...) which somehow screwed up the monotlith and made it start crashing.

This statement makes no sense at all.

> Obviously, it was a 100% stupid idea (...)

I saw no stupid idea in any of your statements.

You stated the legacy codebase was crap, and that a team member took up to himself to do the strangler's vine thing and gradually peel responsibilities out of the monolith. What leads you to believe this is stupid?

[atleta]

> Why do you feel this is relevant, let alone detrimental to the idea of microservices? It looks to me that it's one of the primary positive traits.

I did explain at the end of the sentence, why it was relevant: because rewriting in JS would have also meant switching to a completely new language. Which the team, as a whole had less experience with and which would have made it a complete rewrite without being able to reuse anything.

> This statement makes no sense at all.

I'm sorry you didn't understand it. If you asked a clarifying question I may have been able to explain.

> You stated the legacy codebase was crap,

No, I said one team member stated that the code base was crap. I didn't say it was crap. I didn't evaluate it, because I don't know (and don't like) PHP and because I didn't have to work with it anyway. The other two team members said the code was of acceptable quality for them to work on.

> and that a team member took up to himself to do the strangler's vine thing and gradually peel responsibilities out of the monolith. What leads you to believe this is stupid?

Because this is not what was going to happen and because they had been working on the thing for about 2 years back then without releasing it and because their planned release would have been in about 2 months. It's almost never about only the technology, these products and services serve a business purpose or provide value through other means. If it didn't matter, you can of course take all the time in the world, start over 10 times just to come up with what you think the best solution is for the problem. It's totally fine, this is the artistic approach. If you follow the engineering approach then you have to factor in the time and the investment too. Because in that case you have to deliver value.

Regarding peeling away gradually, this wasn't really what I said. The person pushing for the microservices solution said they needed to rewrite. So stop the world, rebuild the thing without having a working system during that period. (Because they didn't have a working system to start with.)