[vlovich123]

You can do that but you keep missing that you’re no longer a true microservice as originally defined and envisioned, which is that you can deploy the service independently under local control.

Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services.

OP is correct that you are indeed now in a weird hybrid monolith application where it’s deployed piecemeal but can’t really be deployed that way because of tightly coupled dependencies.

Be ready for a blog post in ten years how they broke apart the monolith into loosely coupled components because it was too difficult to ship things with a large team and actually have it land in production without getting reverted to an unrelated issue.

[GeneralMayhem]

Internal and external have wildly different requirements. Google internally can't update a library unless the update is either backward-compatible for all current users or part of the same change that updates all those users, and that's enforced by the build/test harness. That was an explicit choice, and I think an excellent one, for that scenario: it's more important to be certain that you're done when you move forward, so that it's obvious when a feature no longer needs support, than it is to enable moving faster in "isolation" when you all work for the same company anyway.

But also, you're conflating code and services. There's a huge difference between libraries that are deployed as part of various binaries and those that are used as remote APIs. If you want to update a utility library that's used by importing code, then you don't need simultaneous deployment, but you would like to update everywhere to get it done with - that's only really possible with a monorepo. If you want to update a remote API without downtime, then you need a multi-phase rollout where you introduce a backward-compatibility mode... but that's true whether you store the code in one place or two.

[vlovich123]

The whole premise of microservices is loose coupling - external just makes it plainly obvious that it’s a non starter. If you’re not loosely coupling you can call it microservices but it’s not really.

Yes I understand it’s a shared library but if updating that shared library automatically updates everyone and isn’t backward compatible you’re doing it wrong - that library should be published as a v2 or dependents should pin to a specific version. But having a shared library that has backward incompatible changes that is automatically vendored into all downstream dependencies is insane. You literally wouldn’t be able to keep track of your BOM in version control as it obtains a time component based on when you built the service and the version that was published in the registry.

[GeneralMayhem]

> if updating that shared library automatically updates everyone and isn’t backward compatible you’re doing it wrong that library should be published as a v2 or dependents should pin to a specific version

...but why? You're begging the question.

If you can automatically update everyone including running their tests and making any necessary changes to their code, then persisting two versions forever is a waste of time. If it's because you can't be certain from testing that it's actually a safe change, then fine, but note that that option is still available to you by copy/pasting to a v2/ or adding a feature flag. Going to a monorepo gives you strictly more options in how to deal with changes.

> You literally wouldn’t be able to keep track of your BOM in version control as it obtains a time component based on when you built the service

This is true regardless of deployment pattern. The artifact that you publish needs to have pointers back to all changes that went into it/what commit it was built at. Mono vs. multi-repo doesn't materially change that, although I would argue it's slightly easier with a monorepo since you can look at the single history of the repository, rather than having to go an extra hop to find out what version 1.0.837 of your dependency included.

> the version that was published in the registry

Maybe I'm misunderstanding what you're getting at, but monorepo dependencies typically don't have a registry - you just have the commit history. If a binary is built at commit X, then all commits before X across all dependencies are included. That's kind of the point.

[vlovich123]

> ...but why? You're begging the question. If you can automatically update everyone including running their tests and making any necessary changes to their code, then persisting two versions forever is a waste of time.

I’m not begging the question. I’m simply stating what loose coupling looks like and the blog post is precisely the problem of tight coupling. If you have multiple teams working on a tightly coupled system you’re asking for trouble. This is why software projects inevitably decompose against team boundaries and you ship your org chart - communication and complexity is really hard to manage as the head count grows which is where loose coupling helps.

But this article isn’t about moving from federated codebases to a single monorepo as you propose. They used that as an intermediary step to then enable making it a single service. But the point is that making a single giant service is well studied and a problem. Had this constantly at Apple when I worked on CoreLocation where locationd was a single service that was responsible for so many things (GPS, time synchronization of Apple Watches, WiFi location, motion, etc) that there was an entire team managing the process of getting everything to work correctly within a single service and even still people constantly stepped on each other’s toes accidentally and caused builds that were not suitable. It was a mess and the team that should have identified it as a bottleneck in need of solving (ie splitting out separate loosely coupled services) instead just kept rearranging deck chairs.

> Maybe I'm misunderstanding what you're getting at, but monorepo dependencies typically don't have a registry - you just have the commit history

I’m not opposed to a monorepo which I think may be where your confusion is coming from. I’m suggesting slamming a bunch of microservices back together is a poorly thought out idea because you’ll still end up with a launch coordination bottleneck and rolling back 1 team’s work forces other teams to roll back as well. It’s great the person in charge got to write a ra ra blog post for their promo packet. Come talk to me in 3 years with actual on the ground engineers saying they are having no difficulty shipping a large tightly coupled monolithic service or that they haven’t had to build out a team to help architect a service where all the different teams can safely and correctly coexist. My point about the registry is that they took one problem - a shared library multiple services depend on through a registry depend on latest causing problems deploying - and nuked it from orbit using a monorepo (ok - this is fine and a good solution - I can be a fan of monorepos provided your infrastructure can make it work) and making a monolithic service (probably not a good idea that only sounds good when you’re looking for things to do).

[necovek]

> I’m not begging the question. I’m simply stating what loose coupling looks like and the blog post is precisely the problem of tight coupling.

But it is not! They were updating dependencies and deploying services separately, and this led to every one of 140 services using a different version of "shared-foo". This made it cumbersome, confusing and expensive to keep going (you want a new feature from shared-foo, you have to take all the other features unless you fork and cherrypick on top, which makes it a not shared-foo anymore).

The point is that true microservice approach will always lead to exactly this situation: a) you either do not extract shared functions and live with duplicate implementations, b) you enforce keeping your shared dependencies always on very-close-to-latest (which you can do with different strategies; monorepo is one that enables but does not require it) or c) you end up with a mess of versions being used by each individual service.

The most common middle ground is to insist on backwards compatibility in a shared-lib, but carrying that over 5+ years is... expensive. You can mix it with an "enforce update" approach ("no version older than 2 years can be used"), but all the problems are pretty evident and expected with any approach.

I'd always err on the side of having a capability to upgrade at once if needed, while keeping the ability to keep a single service on a pinned version. This is usually not too hard with any approach, though monorepo makes the first one appear easier (you edit one file, or multiple dep files in a single repo): but unless you can guarantee all services get replaced in a deployment at exactly the same moment — which you rarely can — or can accept short lived inconsistencies, deployment requires all services to be backwards compatible until they are all updated with either approach).

I'd also say that this is still not a move to a monolith, but to a Service-Oriented-Architecture that is not microservices (as microservices are also SOA): as usual, the middle ground is the sweet spot.

[wowohwow]

To reference my other comment. This thread is about the nuance of if a dependency on a shared software repository means you are a microservice or not. I'm saying it's immaterial to the definition.

A dependency on an external software repository does not make a microservice no longer a microservice. It's the deployment configuration around said dependency that matters.

[necovek]

What everyone else is saying is that the core value proposition of microservices is that they are independently deployable (which I believe is what you are aiming for as well), which means that there is no tight coupling between them.

If one introduces tight coupling by having a shared library that gets updated in backwards incompatible way and needs to be updated simultaneously in each microservice, you move away from a microservices architecture as your services are not independently deployable anymore.

So in the general case, it is immaterial, but in practice, it can be a mechanism which introduces tight coupling and negates the core value of the microservices architecture.

Here, it was done on purpose as a step to a more monolithic architecture (though it was still only a single service in a larger system, so I'd avoid the "monolith" term).

[smaudet]

> Be ready for a blog post in ten years how they broke apart the monolith into loosely coupled components because it was too difficult to ship things with a large team and actually have it land in production without getting reverted to an unrelated issue.

Some of their "solutions" I kind of wonder how they plan on resolving this, like the black box "magic" queue service they subbed back in, or the fault tolerance problem.

That said, I do think if you have a monolith that just needs to scale (single service that has to send to many places), they are possibly taking the correct approach. You can design your code/architecture so that you can deploy "services" separately, in a fault tolerant manner, but out of a mono repo instead of many independent repos.

[necovek]

They don't have a monolith: they have a service that has a restricted domain of responsibility matched to the team that runs it.

There is nothing magic about their queue service, and it seems correctly tuned to the complexity that they've got to cover: yes, just like most queue implementations, it will get different types of messages (events). If anything, their previous implementation was too complex which caused lots of waste.

With hindsight, they should have evolved their original architecture into exactly what they pivoted to now: better fault tolerance in "processors" of different types.

I would hope that my general rule of "only solve exactly the problem you have in front of you" would have avoided the approach they took, but engineers love to abstract away things and introduce indirection layers and add accidental complexity that way. And ofc, "microservices great, me want microservices" too :)

Again, I am not saying this as a slight: I believe many of us have learned the limits of microservices by, well, living through them :) And now we tune our abstraction layers differently.

[smaudet]

> They don't have a monolith: they have a service that has a restricted domain of responsibility matched to the team that runs it.

Except, for lack of a better definition, that is a monolith.

Which there's nothing wrong with one if that's what you need.

> I would hope that my general rule of "only solve exactly the problem you have in front of you"

True. Which was the issue with everyone jumping on the microservice train, most of it was about solving problems nobody had.

When you really need an independent service, go build an independent service. Call them micro if you like (again, no good definition for what microservice or monolith actually mean).

[necovek]

There are better definitions. Monolith would be a full product (as we are talking Twilio, their main offering) in a single, tightly coupled codebase which does data passing mostly by directly calling appropriate code paths.

Service-oriented architecture is where you decouple parts of the full architecture into independent subsystems that communicate over opaque interfaces and without ability to break those boundaries.

Microservices architecture is a SOA taken in the direction of keeping them as small as possible, which can be very small!

[vlovich123]

Why issue isn’t with the monorepo but slamming all the microservices into a single monolithic service (the last part of the blog post).

[dmoy]

> Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services.

Internal Google services: *sweating profusely*

(Mostly in jest, it's obviously a different ballgame internal to the monorepo on borg)