[ddejohn]

This strikes me as overly cynical.

My current job is working with around 20 other engineers on an extremely bloated and coupled monolith. I'd love to be able to separate myself and my team from others by an agreed-upon interface.

Yes, "agreed-upon" is certainly doing some heavy lifting there, but I don't think it's realistic to expect that

> cross team meetings and project management every time you want to ship a feature

is somehow avoidable in tech?

[irjustin]

> This strikes me as overly cynical.

> My current job is working with around 20 other engineers on an extremely bloated and coupled monolith. I'd love to be able to separate myself and my team from others by an agreed-upon interface.

> Yes, "agreed-upon" is certainly doing some heavy lifting there, but I don't think it's realistic to expect that

This is overly cynical to you likely because you haven't experienced maintaining micro-services in the long term. Being able to "break away" with a common interface is literally the microservice tag line. It's completely true too. Standing up a service w/ an interface is incredibly fast, rewarding and watching it hum is beautiful.

The problem starts when there's a bug that's upstream of your service. It's not too hard to get all 15+ services running on your laptop, but the problem is the other teams no longer let external members directly deploy the services they maintain (that one time someone from another team deployed a big bug). So now you've got to get PR approval and seemingly no one wants to review your bug. So you ask for time from the Product Owner and it gets forgotten because they're really busy too. So you go to an eng manager, etc etc...

The above scenario plays out SO many times. I've been there in all various forms. As a blunt statement - any team under 30 members, monolith.

[hmeh]

If you have to "get all 15+ services running on your laptop", you've already lost. Every one of our 100+ components maintained by our dozen person team can be run and tested on its own. The only reason we have Docker on our machine is to test the images we deploy (just to make sure they build and run).

You have experienced what happens when web developers cosplay as software architects. That's when you get "microservices". It's a travesty, and it has ruined an entire generation of software developers who now believe that partitioning and autonomy are just myths.

So, as a blunt statement - your experience is typical. That is, it's average. That is, you experienced an average team doing average work misguided by average bad advice. It says absolutely nothing about software design. It only speaks to the fat part of the bell curve doing what it always does.

[irjustin]

> So, as a blunt statement - your experience is typical. That is, it's average.

Definitely a fair judgement and would agree that this constitutes poor architecting on many many companies' parts including ones that I've been a part of.

> Every one of our 100+ components maintained by our dozen person team can be run and tested on its own.

Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team?

More directly, what's wrong with a monolith at the dozen person team size?

[hmeh]

> Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team?

> More directly, what's wrong with a monolith at the dozen person team size?

Great question. It's honestly a hard question to answer because the real answer requires tacit knowledge, but I'll try anyway because you asked. I'll link a couple articles of longer form writing that may help to answer as well.

The benefit ultimately is productivity. That productivity comes from autonomy, which comes from partitioning.

Each component can be worked, tested, and deployed on its own. We never need to spin up more than one at a time, which means any time we are working on our code, we are typically working on 1/400th of it, and we know we are because we have 400 separate GitHub repositories. We can't accidentally get peanut butter in our mayonaise. Each part is independently built and tested.

When it's time to upgrade Rails, we can upgrade one application. This is 1/20th of our Rails code. That's a smaller batch. We know from basic flow principles that (see: Principles of Product Development Flow, Lean, etc.) that small batches are better for productivity.

We rarely have merge conflicts. We don't do pull requests. We branch when we need to. We are often working on a brand new, never been deployed project. We don't have our tests running in CI. Our tests run in less than a second in most projects on our laptops. With those that have UI, our tests typically run in less than 30 seconds which include thorough Capybara (UI Interaction) testing.

When we are working in any application, we only need to consider its direct efferents when making a change to any interfaces it exposes. We can trace those changes through easily when we need to make them. Because we practice software design, we don't often need to make large sweeping changes, but when we do, we can do them methodically without disruption.

The "worst" part is that some things are tedious. But we can automate tedious things. We can invent around tedious things. We used to have to deploy 15 web applications any time we had a style change. Now we deploy one application that supplies the CSS for all our applications via SSI (server-side-include). If we need to deploy 70 back end components, we can script that (with manual verification at each step). We can even spread that out across a couple team members and do it as an ensemble. It takes about an hour to deploy everything. We maybe do this once a month or so (when something highly afferent changes -- and guess what, we structure our design so that afferent things change as infrequently as possible).

Most things we work on we work until they are "complete". That is, they rarely change after that. We have components that have been in production for 3 years and haven't been substantially touched aside from Ruby upgrades and the like. They are still perfectly in control.

The list goes on and on.

Here are a few articles, feel free to poke around the others and ask any follow up questions.

Partitions and Compositions: https://github.com/aaronjensen/software-development/blob/mas...

The Mythical Monolith: https://github.com/aaronjensen/software-development/blob/mas...

[hw]

Great that it works for you and your particular project.

Without knowing the details if your project and team, I’d take a monolith any day over 400 microrepos and the 70+ backend microservices. Our customers don’t care if it’s microservices or monoliths as long as it works and meets SLAs, so the question really is do I want to upgrade ruby or rails 400 times and deploy 100 services or just do it once in my monolith and deploy once?

Need to upgrade a gem that’s used in 100 of those repos to address a CVE - no thanks. Worse, 10 of those repos were owned by someone who left the company a year ago and since there were no PRs, no one knows how to work in the repo or understand what it does.

[hmeh]

That isn't actually the question. I know it's not the question because we're capable of it, it is infrequent, and it takes on the order of single digit person hours a month for that type of concern for our particular project. Also, I didn't say 400 anything but repos. See some of my other comments for what those repos consist of.

> 10 of those repos were owned by someone who left the company a year ago

That's not how it works. Repos aren't "owned" by someone. Each repo stands on its own, is clearly testable on its own. Every one of our teammates is capable of updating every one of our projects because there is consistency between them.

If upgrading Rails or any gem scares you, you have other problems to deal with. For me personally (I'm relatively fast on this team) I can update Rails in all 20 of our Rails applications in maybe 30 minutes unless Rails caused a setback (this is somewhat frequent between propshaft and turbo). If Rails caused a setback, we have two dependencies (rails-application-operational and rails-application-development) that we can either include a patch or pin Rails. All told, it's a quick process.

You know what we don't have to spend time doing? Tracing callbacks across 30 entangled ActiveRecord models. Or anything else that comes from entanglement. We also don't need to entertain palliatives like packwerk and the like.

[the_gipsy]

> we have 400 separate GitHub repositories

Do you think this sounds good?

> We can't accidentally get peanut butter in our mayonaise

Hint: you have 400 different jars of peanut butter with mayonnaise.

[hmeh]

How do you know that? Hint: you don’t. You’re not the first to tell us what we do is impossible, and you won’t be the last.

By the way, I know how it sounds. And I know why a person would fear it. And, I know why they’d be incorrect in their assessment, because, unlike them, I have significant material experience in both camps.

[Too]

> Each part is independently built and tested.

> We don't have our tests running in CI.

> It takes about an hour to deploy everything.

Eh? 400 components without any integration testing?? Who glues the sum of the parts together into a whole? This can’t be a single application. Sounds more like many completely independent applications to me.

[hmeh]

Who said we don't do integration testing? We likely don't do it in the way that you would imagine, because we don't need to. We also don't call it integration testing, we just call it testing. We have a test environment where we deploy to to do final inspection. It is a common occurrence that we deploy a cohesive set of new components for the first time and they all work together, because each component is tested in isolation including integration with its afferents. Each afferent (library or other component) exposes diagnostic substitutes and we use dependency injection. You can read more about that here: http://docs.eventide-project.org/user-guide/useful-objects.h...

Also, I never said 400 components -- we aren't there, yet. It's ~100 components. It's just 400 repositories. 100 of those are deploy projects (they simply contain scripts, a dockerfile, helm values, etc.) most of which is boilerplate.

[irjustin]

Thanks for the longer writeup, I appreciate it. It'll take me some time to digest the concepts.

[hmeh]

You are welcome. Feel free to reach out at any time with follow up questions. GitHub discussions are probably easiest, but I’ll try to check here too.

[rezonant]

I don't intend this to be mean, but in some parts this reads as satire.

[mewpmewp2]

Yes, thought so as well. "Instead of a single search and replace we get to a flow state and create an automated script to do it for all our repositories. Due to flow state productivity grains are immense. Of course we will manually check 400 times if the script did correct work, but we enjoy that each time."

[hmeh]

Well, there is Poe's law. You can call my views and our team's culture extreme. We believe it is extreme from a statistical perspective. There's a reason that our team has the tenure that it has. What we are doing isn't out there typically.

But, I would challenge you to challenge anything that you saw as satire and try and wrap your head around how a sane, rational person could, with a straight face say "that's not a problem" and be absolutely right. Feel free to call something out specifically to me, and I will back it up. I'll warn you that you may still not agree with me, because much of what we do requires tacit knowledge to see the benefit of. And once you do, it's self-evident. A person must be studied in Lean and design principles and have seen the trajectory of several software projects over the course of many years.

See if you can understand what are the actual problems in software. What causes projects to fail and teams to have to hire 100s of developers to maintain a semblance of the productivity they had for their first year or two of development.

Try to record your setbacks in a daily work log (we all do this) and address root causes. Do this for 10-20 years and see where you end up.

[kamikaz1k]

Just to clarify, what are components referring to? Are they independent web services?

And what is the total size of the developers you worked with? Are the 12 just your team or the entire engineering org?

Edit: found from another comment

> We don’t call them microservices, because we aren’t web developers pretending to do architecture and recreating the mistakes of DCOM, CORBA, and Web Services. Most of them are autonomous event sourced components. The others are stand alone web applications that are stitched together with Nginx routing and SSI.

[hmeh]

Components are either autonomous event sourced back end components (using Eventide) or independent web applications that are combined with Nginx routes and SSI. The UI is (mostly) server-rendered Rails. The users have no idea they are hitting 20 or so different web applications, but the developers sure feel the productivity boost of every application being a small application that is (relatively) independent from one another.

A dozen or so is the entire software development organization.

Edit: Indeed! If you have any other questions, ask away. Most people think what we do isn't actually possible, but it is.

[8n4vidtmkvmk]

Standing up a service across the globe is non-trivial even with world class tools. You still run into issues like services not being properly co located with other services they need to call so you end up paying for inter continental hops. And cold connections because the new service doesn't have much traffic yet. And maybe DB leaders are only in North America so Asia has terrible perf.

[roenxi]

> I'd love to be able to separate myself and my team from others by an agreed-upon interface.

Don't doubt it, but that is what the grandparent was going to - you don't want to be held accountable for the success of the overall product, you want a defined contract that you maintain and the value-add of that is someone else's problem.

The dysfunction of a business is made up of individuals doing what makes the most sense for them locally. It isn't unreasonable to want microservices, it just happens that the incentives that make developers like microservices also tend towards systems that lack accountability for achieving system-level outcomes.

[jpk]

Isn't this precisely what leadership is for? If you've implemented an architecture that mirrors your org chart, then the interfaces aren't just in software, they exist between managers as well. System-level success depends on the managers and the directors above them to steer the boat in a way that places accountability on the teams and components of the architecture necessary to ship a feature, or whatever.

[onion2k]

Isn't this precisely what leadership is for?

If your devs are just implementing whatever system design is coming from your tech leads, and having no opportunity to have an input or their own, you'll find you have very unhappy devs who don't hang around very long. That churn will make it impossible to build a decent system.

[jpk]

If your devs are just implementing whatever they want, with no strong design from your tech leads, and leaving no room for input from other teams, you'll find you have very unhappy customers who don't hang around very long... etc.

It's a balance.

[kaashif]

> If you've implemented an architecture that mirrors your org chart

In my experience this is almost invariably what happens. I would replace the if with a when.

Wow, you're telling me our three teams want to write three separate services? No way.

And the teams in New York want to use one technology while the teams in San Francisco want to use something else? Say it ain't so!

[Nextgrid]

> I'd love to be able to separate myself and my team from others by an agreed-upon interface.

My experience suggests that speccing out such an interface well is a tremendous amount of work, and if you get it wrong (you most likely will) or if it needs to evolve it will generate lots of work since now it's set in stone and very hard to change so you're likely going to pay "interest" by working around its shortcomings for the foreseeable future.

[twhitmore]

Difficulty & cost evolving the interface between microservices can seriously hinder improving the actual product.

I've seen the misguided adoption of microservices affect a couple of companies & products.

[the_gipsy]

The interface will evolve, and in fact that is the hard part of software development.