[irjustin]

> So, as a blunt statement - your experience is typical. That is, it's average.

Definitely a fair judgement and would agree that this constitutes poor architecting on many many companies' parts including ones that I've been a part of.

> Every one of our 100+ components maintained by our dozen person team can be run and tested on its own.

Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team?

More directly, what's wrong with a monolith at the dozen person team size?

[hmeh]

> Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team?

> More directly, what's wrong with a monolith at the dozen person team size?

Great question. It's honestly a hard question to answer because the real answer requires tacit knowledge, but I'll try anyway because you asked. I'll link a couple articles of longer form writing that may help to answer as well.

The benefit ultimately is productivity. That productivity comes from autonomy, which comes from partitioning.

Each component can be worked, tested, and deployed on its own. We never need to spin up more than one at a time, which means any time we are working on our code, we are typically working on 1/400th of it, and we know we are because we have 400 separate GitHub repositories. We can't accidentally get peanut butter in our mayonaise. Each part is independently built and tested.

When it's time to upgrade Rails, we can upgrade one application. This is 1/20th of our Rails code. That's a smaller batch. We know from basic flow principles that (see: Principles of Product Development Flow, Lean, etc.) that small batches are better for productivity.

We rarely have merge conflicts. We don't do pull requests. We branch when we need to. We are often working on a brand new, never been deployed project. We don't have our tests running in CI. Our tests run in less than a second in most projects on our laptops. With those that have UI, our tests typically run in less than 30 seconds which include thorough Capybara (UI Interaction) testing.

When we are working in any application, we only need to consider its direct efferents when making a change to any interfaces it exposes. We can trace those changes through easily when we need to make them. Because we practice software design, we don't often need to make large sweeping changes, but when we do, we can do them methodically without disruption.

The "worst" part is that some things are tedious. But we can automate tedious things. We can invent around tedious things. We used to have to deploy 15 web applications any time we had a style change. Now we deploy one application that supplies the CSS for all our applications via SSI (server-side-include). If we need to deploy 70 back end components, we can script that (with manual verification at each step). We can even spread that out across a couple team members and do it as an ensemble. It takes about an hour to deploy everything. We maybe do this once a month or so (when something highly afferent changes -- and guess what, we structure our design so that afferent things change as infrequently as possible).

Most things we work on we work until they are "complete". That is, they rarely change after that. We have components that have been in production for 3 years and haven't been substantially touched aside from Ruby upgrades and the like. They are still perfectly in control.

The list goes on and on.

Here are a few articles, feel free to poke around the others and ask any follow up questions.

Partitions and Compositions: https://github.com/aaronjensen/software-development/blob/mas...

The Mythical Monolith: https://github.com/aaronjensen/software-development/blob/mas...

[hw]

Great that it works for you and your particular project.

Without knowing the details if your project and team, I’d take a monolith any day over 400 microrepos and the 70+ backend microservices. Our customers don’t care if it’s microservices or monoliths as long as it works and meets SLAs, so the question really is do I want to upgrade ruby or rails 400 times and deploy 100 services or just do it once in my monolith and deploy once?

Need to upgrade a gem that’s used in 100 of those repos to address a CVE - no thanks. Worse, 10 of those repos were owned by someone who left the company a year ago and since there were no PRs, no one knows how to work in the repo or understand what it does.

[hmeh]

That isn't actually the question. I know it's not the question because we're capable of it, it is infrequent, and it takes on the order of single digit person hours a month for that type of concern for our particular project. Also, I didn't say 400 anything but repos. See some of my other comments for what those repos consist of.

> 10 of those repos were owned by someone who left the company a year ago

That's not how it works. Repos aren't "owned" by someone. Each repo stands on its own, is clearly testable on its own. Every one of our teammates is capable of updating every one of our projects because there is consistency between them.

If upgrading Rails or any gem scares you, you have other problems to deal with. For me personally (I'm relatively fast on this team) I can update Rails in all 20 of our Rails applications in maybe 30 minutes unless Rails caused a setback (this is somewhat frequent between propshaft and turbo). If Rails caused a setback, we have two dependencies (rails-application-operational and rails-application-development) that we can either include a patch or pin Rails. All told, it's a quick process.

You know what we don't have to spend time doing? Tracing callbacks across 30 entangled ActiveRecord models. Or anything else that comes from entanglement. We also don't need to entertain palliatives like packwerk and the like.

[the_gipsy]

> we have 400 separate GitHub repositories

Do you think this sounds good?

> We can't accidentally get peanut butter in our mayonaise

Hint: you have 400 different jars of peanut butter with mayonnaise.

[hmeh]

How do you know that? Hint: you don’t. You’re not the first to tell us what we do is impossible, and you won’t be the last.

By the way, I know how it sounds. And I know why a person would fear it. And, I know why they’d be incorrect in their assessment, because, unlike them, I have significant material experience in both camps.

[outsomnia]

This whole business has had few conceptual ideas that stand apart from fashion and stood the tests of time. A lot of people look down on C now for various reasons, but it's still extremely useful and going strong 50+ years later. Even though I'm older than C itself in the last year or two using it intensively, I found new insights in how to arrange things and what to avoid to make things more robust and maintainable.

It's not likely that hundreds of "microrepos" on github and even rails are The Last Word that people will eventually converge on and be using into the 2100s. It sounds like each repo is its own standalone thing with its own deps. There are advantages and disadvantages to maintaining this kind of thing, some of the disadvantages will only become clear as the deps evolve and mutate. Although you may feel you are in a local optimum with this architecture in 2023, ahead of the game, energized by your belief in it, you probably are not finished evolving your understanding even on the scale of one or five years.

One way or another almost everything in this business is ephemeral, disposable and will be disposed of in favour of something else.

[hmeh]

Do you know what does stand the test of time? Design principles that are based on the immutable laws of physics. There's a reason that I can quote Plato to support the notion of partitioning. There's a reason that the basic principles are used in the design of every single significant thing in the world. There's a reason Linux has as many modules as it has (for them, a single monorepo works, but multiple works too -- see any library or application that isn't in the primary monorepo but is distributed via a package manager).

Also, no one said "microrepos". Don't put the current fad's fixation on what I am describing. I'm simply describing structural design. Does thing X and thing Y need common behavior? Is it actually generalized behavior? That's what we have called a "library" since we've been building non-trivial applications out of software.

Are there two disparate business processes that don't need to share states, and you know they don't need to share states because you've done the design work? Those are two separate components. Pub/Sub exists for collaboration between those components as necessary.

I'm not trying to say that our current technique for all things is the end-all-be-all. That would deny the continuous improvement mindset (i.e., the relentless pursuit of perfection) that got us where we are. But, there are things that won't change. Smaller batches will always be (generally) easier to manage than larger. Taking something that has high afferent coupling and adding specializations to it will always be a mistake (See Fat Model, Skinny Controller).

The reality is that there are software design mistakes that are knowably mistakes. They've been known for some time as well and can be measured. Those are the things that we avoid and those are the principles that have guided where we are.

I know this sounds highfalutin, but there is someone on our team that has been doing this for over 30 years (by "this", I mean continuous improvement/seeking perfection) and has studied under many of the greats in our industry. We aren't just making this stuff up.

[sixstringtheory]

Nobody is denying partitioning is helpful. It’s the implementation details of that partitioning that cause the debate. IMO using repos as a partition boundary is an abuse of git. And IMO invoking Plato to make a point is not helpful. Argumentum ad antiquitatem. This goes for the “30 years” comment as well: it’s entirely conceivable that they’ve been doing things inefficiently for 30 years. More likely that there is such a huge sunk cost that they are heavily invested in justifying it.

[jrs235]

Are you working with Robert C. Martin?!

[sixstringtheory]

As someone who has worked in monoliths and microservices, monorepos and github galaxies, I think it would be better if we could converge on a monorepo managed by something like Bazel where you can just create a new build target that lists its dependencies, and can work directly with the dependency graph via its querying mechanism. Using repositories to encapsulate dependencies and build scripts I think is much too heavy of an abstraction and leads to poor discoverability. When I have to deal with that I usually wind up treating it like a monorepo anyways by creating automation to check out all repos and manage syncing them.

[hmeh]

> When I have to deal with that I usually wind up treating it like a monorepo anyways by creating automation to check out all repos and manage syncing them.

Yes! Absolutely. With one key difference: You can't commit to multiple repos at the same time. You can certainly have them all checked out locally and search across them all, and we provide scripts to do that. Example from the Eventide project: https://github.com/eventide-project/contributor-assets

This is an open source project. Our project is based on many of the ideas and techniques. They've held up for 7 years or so.

[Too]

> Each part is independently built and tested.

> We don't have our tests running in CI.

> It takes about an hour to deploy everything.

Eh? 400 components without any integration testing?? Who glues the sum of the parts together into a whole? This can’t be a single application. Sounds more like many completely independent applications to me.

[hmeh]

Who said we don't do integration testing? We likely don't do it in the way that you would imagine, because we don't need to. We also don't call it integration testing, we just call it testing. We have a test environment where we deploy to to do final inspection. It is a common occurrence that we deploy a cohesive set of new components for the first time and they all work together, because each component is tested in isolation including integration with its afferents. Each afferent (library or other component) exposes diagnostic substitutes and we use dependency injection. You can read more about that here: http://docs.eventide-project.org/user-guide/useful-objects.h...

Also, I never said 400 components -- we aren't there, yet. It's ~100 components. It's just 400 repositories. 100 of those are deploy projects (they simply contain scripts, a dockerfile, helm values, etc.) most of which is boilerplate.

[irjustin]

Thanks for the longer writeup, I appreciate it. It'll take me some time to digest the concepts.

[hmeh]

You are welcome. Feel free to reach out at any time with follow up questions. GitHub discussions are probably easiest, but I’ll try to check here too.

[rezonant]

I don't intend this to be mean, but in some parts this reads as satire.

[mewpmewp2]

Yes, thought so as well. "Instead of a single search and replace we get to a flow state and create an automated script to do it for all our repositories. Due to flow state productivity grains are immense. Of course we will manually check 400 times if the script did correct work, but we enjoy that each time."

[hmeh]

Usually, when you put quotes around something, it's considered proper to actually be quoting someone instead of trying your best to make them sound like a clown.

But, why don't I go ahead and give you a real quote.

Of our 400+ repos, 100 or so of them are deploy projects. The only time they need to change in batch is if one of our boilerplate scripts changes. This is infrequent, but when it does happen, we have a "project-scripts" repository that contains our boilerplate scripts as well as scripts to update them across all of our repositories. This doesn't always require individual verification, but when it does, it's typically a matter of installing dependencies and starting the application.

Many text editors are capable of doing "single search and replace" across multiple repositories. All of our repos are in a single project directory, so we have a root, just like anyone with a monorepo does. Rarely does that "single search and replace" touch more than a small handful of projects. So, except in the extreme case, you are only running a fraction of the tests or verifications when you make one of these "search and replace" changes. Compare this to a monorepo, where, unless you have specialized tooling to slice your test suite while considering your package dependency tree, you end up needing to run all of your tests.

If you are responsible and your project is in control, you run the tests on your laptop first, and then (typically) you push your commit and it runs in CI. Actually, more typically, you push your commit to a branch, open a pull request, have it run in CI, have it reviewed, then merge it, where it runs in CI again. Each time it ran in CI, it ran your entire test suite (usually).

We don't do most of that. Most of what every team takes for granted, we just don't do. So, when someone imagines having to do their typical process for getting a change integrated 100 times, they are right to fear that. But, they are creating a false equivalence. We don't do that, because we've eliminated all of that waste from our process. What we do 100 times (again, the once or twice a month that we do it), it takes a fraction of the offort what most teams do once to get a single change in.

There's a reason that Toyota ate the west's lunch when it comes to manufacturing. They understand just-in-time (one-piece flow) and small batch sizes. They know how to address problems at their root and systematically eliminate waste.

Most in the software industry know how to do one thing: apply the thing the latest dev celebrity said in their latest tweet or blog post.

Here's the difference between me and that celebrity: I'll tell you straight up not to do what we do. I'll tell you straight up it will take you years with guidance to get anywhere close to what we do. I'm just telling you it's possible and we don't actually need to live in software shanty towns our whole lives.

[elktown]

Someone got to say it so I will. Most of your comments comes across like you've gone off the rails and started replacing competency, not to mention humility, with narcissism. Being so cocksure and backing it up with statements that you need to be as brilliant as you to "understand" is such a laughingly large red flag that I'm not surprised people interpret it as satire. I really hope your colleagues are onboard with this and that this isn't your own dogmatic crusade dragging them along.

[jrs235]

I don't read cockiness or narcissism in his posts. Having just started reading Clean Architecture by Robert C. Martin two days ago (and hoping to complete it tonight, then planning on re-reading it again), it seems to me that hmeh has learned and PRACTICED what is discussed in that book. I wish I were in their shoes!

[hmeh]

> Someone got to say it so I will. Most of your comments comes across like you've gone of the rails and started replacing competency, not to mention humility, with narcissism. Being so cocksure and backing it up with statements that you need to be as brilliant as you to "understand" is such a laughingly large red flag that I'm not surprised people interpret it as satire.

Where did I say that, exactly? I’ve said nothing about intellect. I’ve mentioned tacit knowledge multiple times, because I have enough experience to recognize how ludicrous a lot of this sounds. It sounded that way to me at first until I experienced it and saw the pieces fit together. That’s how tacit knowledge works, especially when it is counter to average knowledge. It’s the same reason American industry smashed Japanese cars in demonstration instead of listening to them when they said large batches were bad (I’m simplifying here somewhat)

And which part, specifically sounds incompetent? I’m happy to discuss that.

By the way, it's decidedly difficult to challenge a common belief, dare I say, orthodoxy, without sounding insane, confident, or even cocky. Could I be better at it? Of course, and I work at it, but I also am not here to make friends. I'm here to provide a perspective that I believe is sorely lacking from the development community and is drowned out by the orthodoxy, its acolytes, and its beneficiaries.

> I really hope your colleagues are onboard with this and that this isn't your own dogmatic crusade dragging them along.

Given their tenure on the team and their other options, it’s a pretty safe bet. It’s not my crusade, btw, it’s our teams goal to manage a relatively complex project with a relatively small team.

Also, someone has to say it, so I will. You have now dropped this conversation, which was about the work, into ad hominem and straw man attacks. That’s one of the worst part about this industry: people’s inability to debate and discuss without attacking people or glorifying celebrities.

[hmeh]

Well, there is Poe's law. You can call my views and our team's culture extreme. We believe it is extreme from a statistical perspective. There's a reason that our team has the tenure that it has. What we are doing isn't out there typically.

But, I would challenge you to challenge anything that you saw as satire and try and wrap your head around how a sane, rational person could, with a straight face say "that's not a problem" and be absolutely right. Feel free to call something out specifically to me, and I will back it up. I'll warn you that you may still not agree with me, because much of what we do requires tacit knowledge to see the benefit of. And once you do, it's self-evident. A person must be studied in Lean and design principles and have seen the trajectory of several software projects over the course of many years.

See if you can understand what are the actual problems in software. What causes projects to fail and teams to have to hire 100s of developers to maintain a semblance of the productivity they had for their first year or two of development.

Try to record your setbacks in a daily work log (we all do this) and address root causes. Do this for 10-20 years and see where you end up.