[nateburke]

The moment you adopt service based teams with service based managers, say goodbye to engineers caring about working product. Say hello to cross team meetings and project management every time you want to ship a feature.

It's pure vanity for a startup to think they will become the next AWS by adopting hard service-based contracts between teams.

[wavemode]

This viewpoint doesn't translate to reality in my experience. Monolith or not, engineers care about working product if they are incentivized to care about working product (and if they aren't, they don't).

Code is code. Cognitive load is cognitive load. Doesn't matter how you organize it. Unless your company is still very small and simple, there's no single team that is going to be able to understand how the entire system works (and take ownership of every part of it working properly). I've worked in many monoliths where you still had to have "cross team meetings and project management every time you want to ship a feature".

[Nextgrid]

> engineers care about working product if they are incentivized to care about working product

Places that set up the correct incentives are usually very rare. Maybe hedge funds or some finance places which give you a decent chunk of the profits your code makes but that's about it.

Every other place pays you the same "meh" salary and the quickest way up is to job-hop frequently, in which case resume-driven-development takes priority over "working product". That's how you get cargo-cult over-engineering, everything else be damned since you will be gone by the time the consequences emerge anyway.

(also you have "startups" where over-engineering is usually a desired feature as it gives them a justification to grift more VC money to keep their unsustainable business afloat further).

[cncivubyv]

Microservices tends to result in developers thinking their service is the product they're shipping to other developers.

[aniforprez]

Honestly, in my experience there's nothing wrong with this line of thinking as long as you also consider that the product you're shipping to other devs is in service of a much more important and larger product/service in itself. Thinking of your service as a product keeps you thinking of the use cases, the potential errors and the DX of what you're shipping which is a good thing I say as long as you're not building pies in the sky. As with most things, it comes down to good planning

[btown]

The key word here is DX. If a team is exposed to how the DX of their owned service or library enables others in the company to build things that bring smiles to customers' faces, they'll be able to better design for their downstream teams' experiences, finding the right balance of keeping API churn low while building for future flexibility. This is true whether the API boundary is over a network or over a function call.

That said, microservices have the added problem that if you don't get the culture and boundaries set just right from the beginning, it's harder to change. They're good for small domains where boundaries are defined, and many even-early-stage startups have some of those, but it's unlikely that the primary interface for customers is included in that.

[cncivubyv]

This results in people optimizing local maximums, ultimately the real product suffers.

The developers also assume their product is relevant when perhaps it no longer is in the wider context of the business.

[aniforprez]

Yeah that's why I added my conditionals. Obviously the goal is the broader product/service at large

[gedy]

The moment you adopt feature teams with feature managers, say goodbye to engineers caring about working infrastructure. Say hello to cross team meetings and project management every time you want to deal with anything platform related.

I appreciate you qualified as for startups, but worked with plenty of former startups/scaleups that were bogging down rapidly due to infra or scaling issues. Since they were feature factories, no team or PMs were measured or cared about overall system health. This includes "majestic monolith" cough stacks with Rails.

Adding some services were clear wins and needed for things like external APIs, Auth, reporting, etc. - but this is a far cry from AWS microservice hell for simple UI/CRUD features.

[Spivak]

This doesn't ring true for me at all and I've dealt with the dreaded omni-rails stack (or worse the dreaded tomcat springboot stack). Monoliths are far far easier to scale because you have one throat to choke and performance patches / elimination of n+1's spread across the whole app. Also in a monolith unit tests have the power of integration tests in a microservice world. I can actually be sure the boundaries between components behave as I expect and fail loudly if they change in a manner that breaks me. And the last thing in the world I ever want to do for scaling purposes is stick a network call/serialization step between two pieces of my own app because that's a bottleneck waiting to happen.

It feels like a lot of people reach for microservices when they should be reaching for sharding and/or moving to reads to replicas (which you can finally do in Rails thank god).

[ddejohn]

This strikes me as overly cynical.

My current job is working with around 20 other engineers on an extremely bloated and coupled monolith. I'd love to be able to separate myself and my team from others by an agreed-upon interface.

Yes, "agreed-upon" is certainly doing some heavy lifting there, but I don't think it's realistic to expect that

> cross team meetings and project management every time you want to ship a feature

is somehow avoidable in tech?

[irjustin]

> This strikes me as overly cynical.

> My current job is working with around 20 other engineers on an extremely bloated and coupled monolith. I'd love to be able to separate myself and my team from others by an agreed-upon interface.

> Yes, "agreed-upon" is certainly doing some heavy lifting there, but I don't think it's realistic to expect that

This is overly cynical to you likely because you haven't experienced maintaining micro-services in the long term. Being able to "break away" with a common interface is literally the microservice tag line. It's completely true too. Standing up a service w/ an interface is incredibly fast, rewarding and watching it hum is beautiful.

The problem starts when there's a bug that's upstream of your service. It's not too hard to get all 15+ services running on your laptop, but the problem is the other teams no longer let external members directly deploy the services they maintain (that one time someone from another team deployed a big bug). So now you've got to get PR approval and seemingly no one wants to review your bug. So you ask for time from the Product Owner and it gets forgotten because they're really busy too. So you go to an eng manager, etc etc...

The above scenario plays out SO many times. I've been there in all various forms. As a blunt statement - any team under 30 members, monolith.

[hmeh]

If you have to "get all 15+ services running on your laptop", you've already lost. Every one of our 100+ components maintained by our dozen person team can be run and tested on its own. The only reason we have Docker on our machine is to test the images we deploy (just to make sure they build and run).

You have experienced what happens when web developers cosplay as software architects. That's when you get "microservices". It's a travesty, and it has ruined an entire generation of software developers who now believe that partitioning and autonomy are just myths.

So, as a blunt statement - your experience is typical. That is, it's average. That is, you experienced an average team doing average work misguided by average bad advice. It says absolutely nothing about software design. It only speaks to the fat part of the bell curve doing what it always does.

[irjustin]

> So, as a blunt statement - your experience is typical. That is, it's average.

Definitely a fair judgement and would agree that this constitutes poor architecting on many many companies' parts including ones that I've been a part of.

> Every one of our 100+ components maintained by our dozen person team can be run and tested on its own.

Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team?

More directly, what's wrong with a monolith at the dozen person team size?

[hmeh]

> Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team?

> More directly, what's wrong with a monolith at the dozen person team size?

Great question. It's honestly a hard question to answer because the real answer requires tacit knowledge, but I'll try anyway because you asked. I'll link a couple articles of longer form writing that may help to answer as well.

The benefit ultimately is productivity. That productivity comes from autonomy, which comes from partitioning.

Each component can be worked, tested, and deployed on its own. We never need to spin up more than one at a time, which means any time we are working on our code, we are typically working on 1/400th of it, and we know we are because we have 400 separate GitHub repositories. We can't accidentally get peanut butter in our mayonaise. Each part is independently built and tested.

When it's time to upgrade Rails, we can upgrade one application. This is 1/20th of our Rails code. That's a smaller batch. We know from basic flow principles that (see: Principles of Product Development Flow, Lean, etc.) that small batches are better for productivity.

We rarely have merge conflicts. We don't do pull requests. We branch when we need to. We are often working on a brand new, never been deployed project. We don't have our tests running in CI. Our tests run in less than a second in most projects on our laptops. With those that have UI, our tests typically run in less than 30 seconds which include thorough Capybara (UI Interaction) testing.

When we are working in any application, we only need to consider its direct efferents when making a change to any interfaces it exposes. We can trace those changes through easily when we need to make them. Because we practice software design, we don't often need to make large sweeping changes, but when we do, we can do them methodically without disruption.

The "worst" part is that some things are tedious. But we can automate tedious things. We can invent around tedious things. We used to have to deploy 15 web applications any time we had a style change. Now we deploy one application that supplies the CSS for all our applications via SSI (server-side-include). If we need to deploy 70 back end components, we can script that (with manual verification at each step). We can even spread that out across a couple team members and do it as an ensemble. It takes about an hour to deploy everything. We maybe do this once a month or so (when something highly afferent changes -- and guess what, we structure our design so that afferent things change as infrequently as possible).

Most things we work on we work until they are "complete". That is, they rarely change after that. We have components that have been in production for 3 years and haven't been substantially touched aside from Ruby upgrades and the like. They are still perfectly in control.

The list goes on and on.

Here are a few articles, feel free to poke around the others and ask any follow up questions.

Partitions and Compositions: https://github.com/aaronjensen/software-development/blob/mas...

The Mythical Monolith: https://github.com/aaronjensen/software-development/blob/mas...

[hw]

Great that it works for you and your particular project.

Without knowing the details if your project and team, I’d take a monolith any day over 400 microrepos and the 70+ backend microservices. Our customers don’t care if it’s microservices or monoliths as long as it works and meets SLAs, so the question really is do I want to upgrade ruby or rails 400 times and deploy 100 services or just do it once in my monolith and deploy once?

Need to upgrade a gem that’s used in 100 of those repos to address a CVE - no thanks. Worse, 10 of those repos were owned by someone who left the company a year ago and since there were no PRs, no one knows how to work in the repo or understand what it does.

[the_gipsy]

> we have 400 separate GitHub repositories

Do you think this sounds good?

> We can't accidentally get peanut butter in our mayonaise

Hint: you have 400 different jars of peanut butter with mayonnaise.

[Too]

> Each part is independently built and tested.

> We don't have our tests running in CI.

> It takes about an hour to deploy everything.

Eh? 400 components without any integration testing?? Who glues the sum of the parts together into a whole? This can’t be a single application. Sounds more like many completely independent applications to me.

[irjustin]

Thanks for the longer writeup, I appreciate it. It'll take me some time to digest the concepts.

[rezonant]

I don't intend this to be mean, but in some parts this reads as satire.

[kamikaz1k]

Just to clarify, what are components referring to? Are they independent web services?

And what is the total size of the developers you worked with? Are the 12 just your team or the entire engineering org?

Edit: found from another comment

> We don’t call them microservices, because we aren’t web developers pretending to do architecture and recreating the mistakes of DCOM, CORBA, and Web Services. Most of them are autonomous event sourced components. The others are stand alone web applications that are stitched together with Nginx routing and SSI.

[hmeh]

Components are either autonomous event sourced back end components (using Eventide) or independent web applications that are combined with Nginx routes and SSI. The UI is (mostly) server-rendered Rails. The users have no idea they are hitting 20 or so different web applications, but the developers sure feel the productivity boost of every application being a small application that is (relatively) independent from one another.

A dozen or so is the entire software development organization.

Edit: Indeed! If you have any other questions, ask away. Most people think what we do isn't actually possible, but it is.

[8n4vidtmkvmk]

Standing up a service across the globe is non-trivial even with world class tools. You still run into issues like services not being properly co located with other services they need to call so you end up paying for inter continental hops. And cold connections because the new service doesn't have much traffic yet. And maybe DB leaders are only in North America so Asia has terrible perf.

[roenxi]

> I'd love to be able to separate myself and my team from others by an agreed-upon interface.

Don't doubt it, but that is what the grandparent was going to - you don't want to be held accountable for the success of the overall product, you want a defined contract that you maintain and the value-add of that is someone else's problem.

The dysfunction of a business is made up of individuals doing what makes the most sense for them locally. It isn't unreasonable to want microservices, it just happens that the incentives that make developers like microservices also tend towards systems that lack accountability for achieving system-level outcomes.

[jpk]

Isn't this precisely what leadership is for? If you've implemented an architecture that mirrors your org chart, then the interfaces aren't just in software, they exist between managers as well. System-level success depends on the managers and the directors above them to steer the boat in a way that places accountability on the teams and components of the architecture necessary to ship a feature, or whatever.

[onion2k]

Isn't this precisely what leadership is for?

If your devs are just implementing whatever system design is coming from your tech leads, and having no opportunity to have an input or their own, you'll find you have very unhappy devs who don't hang around very long. That churn will make it impossible to build a decent system.

[jpk]

If your devs are just implementing whatever they want, with no strong design from your tech leads, and leaving no room for input from other teams, you'll find you have very unhappy customers who don't hang around very long... etc.

It's a balance.

[kaashif]

> If you've implemented an architecture that mirrors your org chart

In my experience this is almost invariably what happens. I would replace the if with a when.

Wow, you're telling me our three teams want to write three separate services? No way.

And the teams in New York want to use one technology while the teams in San Francisco want to use something else? Say it ain't so!

[Nextgrid]

> I'd love to be able to separate myself and my team from others by an agreed-upon interface.

My experience suggests that speccing out such an interface well is a tremendous amount of work, and if you get it wrong (you most likely will) or if it needs to evolve it will generate lots of work since now it's set in stone and very hard to change so you're likely going to pay "interest" by working around its shortcomings for the foreseeable future.

[twhitmore]

Difficulty & cost evolving the interface between microservices can seriously hinder improving the actual product.

I've seen the misguided adoption of microservices affect a couple of companies & products.

[the_gipsy]

The interface will evolve, and in fact that is the hard part of software development.

[dzikimarian]

Correct question is: why does it take a few teams to deliver feature? This defeats point of entire exercise.

Usually answer is, that division was done by technical role and not business function of the subsystem.

[djur]

In most cases there's going to be substantial overlap between the functionality required for various business functions, and even in differentiated systems the ideal technical domains may not map cleanly to business domains. It's very easy for this to result in multiple teams having half-assed, semi-compatible implementations of the same functionality.

Some places try to address this by creating dedicated "core" or "platform" teams, but there's numerous pitfalls this way as well. Platform teams and product teams necessarily have to justify their budgets in different ways. They have different stakeholders and internal sponsors. This can result in political struggles over headcount. "Brain drain" scenarios where product teams are perennially starved as their strongest ICs are moved to platform. "Slush fund" scenarios where platform teams are regularly "temporarily" reassigned to support feature development. I've seen them both.

The longer you can have your entire engineering team share responsibility for the entire product the better. I am convinced of this. Unless you can truly, completely decouple systems -- making them separate products with dedicated teams -- you run a high risk of having a codebase that is more complex that people still need to understand in its entirety to work effectively. The worst of both worlds.

[dzikimarian]

>Unless you can truly, completely decouple systems -- making them separate products with dedicated teams -- you run a high risk of having a codebase that is more complex that people still need to understand in its entirety to work effectively. The worst of both worlds.

Exactly what I meant. And to make it happen you need to understand underlying business domain. Separate product doesn't have to be huge.

[djur]

For a lot of tech startups the underlying business domain can be nebulous and change significantly over time, and then you end up with substantial organizational debt. Well-established firms can get away with (and benefit from) more differentiation and formal structure because they tend to be able to make longer-term plans.

[ebiester]

It turns out that at a certain point, features start requiring integration in the system. Let's take a hypothetical SaaS that provides workflow management for a domain. Let's say there are about 125 engineers. Workflow can't be 60 engineers, so it's really 12 teams, given teams of 5. One takes the notifications engine and builds emails and slack notifications. They have an API for other teams, but this feature requires a new feature from the notifications system. That's a relatively light touchpoint, but two teams are involved in a dependent relationship.

Now let's say there's a partner integration team. This team provides an external API to ease integration with tools such as Salesforce. This team has a full backlog, and it's a complicated subsystem that doesn't do well as common code. (External APIs also require governance.) So, this feature requires work from the external API. To call this feature finished, this team needs to be able to input and export to other systems, but is not the primary feature of the application. So, you have a second team involved.

Third, you have an internal support team that handles tech support and setup for clients. This team now has work to do to onboard clients.

And there's the team that does the work in the workflow.

As such, this org could need four teams to get a feature done. Now, you could say that such a team could use common code ownership to get things done, but then you have 125 engineers that have to understand a series of systems and not step on each other, systems that they only work in occasionally and change by the time they need to get in again.

I've worked in a common code startup and it works... up to a point. It is definitely slower to have to onboard continuously in other code areas in which you don't have domain experience, even if the documentation is spectacular (and it better be!)

[dzikimarian]

You just hand-waved their entire business as generic "workflow", picked up a few common technical issues (notifications, integration, support), and built teams around these issues. That indeed is going to hurt :-)

[ebiester]

In team topologies, you'd consider them "complicated subsystems" teams. (See https://teamtopologies.com/key-concepts ) They may even be consider stream-aligned teams but they a feature crosses multiple streams to be valuable.

I used a workflow as a common type of SAAS business. My past and current companies could be considered such companies and had significant resources into such streams. Trying to enable the workflow of thousands of companies is a remarkably difficult problem.

But to take an example, look at monday.com as a workflow SAAS business that has a 9 billion dollar market cap. I would be surprised if they didn't have at least one team dedicated to tools for their support team.