[oceanplexian]

I know some of the FAANGs do monorepo (Google being the biggest) but AWS does not.

A monorepo is an organizational mess when trying to manage and transfer ownership across thousands of teams, contain the blast radius of changes, unless you invest a ton of resources into proprietary tooling that requires a bunch of maintenance, since all the open source solutions are terrible at this and the whole data model is built around splitting out individual project repositories. And then after all that effort, why wouldn’t you just use tooling the way it was intended, and the way it’s used in the open source model, so you can partition your CI/CD without a bunch of hacks, and don’t run into bizarre scaling issues with your VCS.

It perplexes me people advocate for this strategy. All I can think is it’s another one of those cargo-cult ideas that everyone is doing because Google did it (So it must be good).

[throwaway894345]

Not having to submit and coordinate PRs across a dozen repos is a pretty tangible benefit.

> unless you invest a ton of resources into proprietary tooling that requires a bunch of maintenance, since all the open source solutions are terrible at this and the whole data model is built around splitting out individual project repositories.

Agree that there's a bunch of tooling needed to operate a monorepo, but there's also a bunch of tooling to sanely manage dozens of "microrepos" as well (when an upstream library changes, update downstream libraries' dependency manifest files to take the new version, run tests, report errors back to upstream, etc). I don't know of any open source tools that manage this problem, but I'm guessing they aren't high-quality if only due to the complex nature of the problem space.

> And then after all that effort, why wouldn’t you just use tooling the way it was intended, and the way it’s used in the open source model, so you can partition your CI/CD without a bunch of hacks, and don’t run into bizarre scaling issues with your VCS.

Because the tooling sucks, as previously mentioned. Many changes require touching many repos, which means coordinating many pull requests and manually changing dependency manifest files and so on.

Ultimately, the "repo" concept is limiting. We need tooling that is aware of dependencies one way or another, and sadly all such open source tooling sucks whether it assumes the relevant code lives in a single repo or across many repos.

[oceanplexian]

> when an upstream library changes, update downstream libraries' dependency manifest files

As someone who's more systems oriented, ideally projects are locked in to a specific versioned dependency, and nothing changes unless a developer of a project explicitly asks for it.

What I've seen is the opposite, someone owns a dependency and is lazy and and wants to perform a breaking operation, and rather than version the change or orchestrate a backwards compatible change, they use mono-repos to "solve" the problem. IMHO it's a bad pattern and leads to a lot of risk.

[throwaway894345]

It's fairly hard to do that in a monorepo world, because (in theory) upstream can't merge anything until downstream tests are passing. Moreover, if you do break something (especially out of laziness), you get reprimanded. And if you really wanted, you could still require downstream approval for upstream changes.

[erik_seaberg]

> when an upstream library changes, update downstream libraries' dependency manifest files

This needs to happen periodically, when we have slack. Doing it continuously adds risks that aren’t really our job to take.

[throwaway894345]

In my experience, if it doesn't happen continuously, it simply doesn't happen at all until something breaks (and then there's a bunch of finger-pointing at upstream even though downstream didn't update). Your first line of defense is that downstream tests are run before anything that affects downstream is merged. The next line of defense is stuff like canary deployments which allow you to minimize blast radius and roll back quickly. Obviously this depends a great deal on your risk regime--if you're SaaS this is probably fine, but if you're embedded this is a non-starter.

[giaour]

Amazon has Brazil versionsets and workspaces, which solve many of the same problems monorepos do. I really liked the way those extra resources let you organize "ad-hoc monorepos" from smaller repositories, but it's infrastructure that I haven't seen elsewhere.

Since leaving Amazon, I've mostly worked with monorepos and wouldn't go back to multirepos without Brazil-style tooling.

[ghoward]

Hey, I'm hoping you can answer some questions for me.

I'm building a build system and a VCS (separately). I want to do it right.

Could you explain to me what Brazil is? Is it the build system? [1] Or is it the VCS that Amazon uses?

If it is the build system, then it appears that versionsets are literally just a list of dependencies with their versions to use for a build. Is that correct? If not, or if you can give me more detail, what are versionsets, exactly?

Also, what are workspaces? Does this quote from one of the comments on the link match?

> A workspace consisted of a version set to track and any packages that were checked out.

[1]: https://gist.github.com/terabyte/15a2d3d407285b8b5a0a7964dd6...

[dastbe]

there's nothing particularly magical and a lot the behaviors are actually similar to bazel, in ways. a version set is a directed acyclic dag of packages linked by dependencies. These edges comprise (more or less)

* normal deps

* compile/test deps (i.e. non-transitive)

* runtime deps

* tool deps (non-transitive, but also non impacting on the closure resolution algorithm)

version sets allow for multiple versions of the same package to exist in them (ex foobar-1.0 and foobar-1.1), which has some benefit but in practice is just painful.

dependencies are defined in a capital-c Config file. when you run brazil, it does a few things

* it resolves all of the tools and makes them available on your path

* it sets up some environment variables

* it invokes your a build command defined in the Config file

Config files can also declare outputs (there's also some canonical outputs), so you can use a query tool to ex.

* get all jars in the runtime closure

* generate a symlink farm of all client sdk configuration files

the results of your build go into a build directory, and when you want to generate the runtime for a particular package, it will symlink together the outputs of all packages in the deps + runtime closure, which you can do on demand.

version sets are updated by building new packages versions into them, which will rebuild the version set to make sure all builds pass. if they do, a new "commit" will be put onto the version set. you can also merge from one version set into another, where you can get packages and their associated dependencies merged in along with a full rebuild.

[ghoward]

Oh, wow, thank you for the detail!

[giaour]

I am not an expert on Brazil, but the link you shared matches my recollection.

It's important to remember that Brazil covers a lot of ground, and many internal tools at Amazon are external to Brazil but rely on it and the way it organizes resources. So I (or others on the internet) may incorrectly call something Brazil if it's part of the larger Brazil ecosystem.

Let's start with a repository and build outwards:

You have some code (let's say it's a Java library) that does something cool. To compile it, you add a Brazil file to the repository root. This file specifies what Brazil packages your code depends on, how it should be built, and what kind of artifact it will produce. Once that file is there, you can run "brazil-build" to produce a Brazil package (which is just a jar with some metadata).

You want to use this library in a web service, so you check out both repositories in a single workspace. Every workspace has a source versionset where it fetches dependencies, but if the code repository for a package is checked out locally, "brazil-build" will build and use the local version instead. You make some changes to the library and web service, then test how they work together by running the web service from within the workspace folder. This ensures that it is using your local modifications to the library repository before those changes have been merged.

Once you're satisfied with the change, you open a PR with a brazil-integrated tool that can show changes to multiple repositories as an atomic change (a "change set"). The CI system for this tool uses a Brazil workspace to make sure your update code packages build together and that all tests pass.

If the PR is approved and you merge to main, there is probably a pipeline watching your repository for changes that proactively rebuilds one or more version sets based on the merged change set. Any package in the version set that depends on your library will be rebuilt using the changes from your change set. So while a versionset is implemented as a list of packages at specific versions, it's best to think of a versionset as more or less equivalent to a monorepo containing all packages on that list, since changes that cross package lines can be built into a versionset in an atomic unit. (This is very helpful if you need to push out a breaking change.) A package can exist in multiple versionsets, which is of course impossible with monorepos.

[ghoward]

Thank you for describing the workflow. This makes it so much easier to understand them!

[dastbe]

very much this. there's also a cultural understanding that libraries/fat clients are equal or worse in terms of maintainability compared to services. equal in the sense that you have to treat them like a service, worse because it's easier to mess up and you don't get to control your rollout strategy.

EDIT: Though while most teams have gone towards many smaller packages for their applications, I suspect that most would be better served by a team level monorepo. That gets you all of the benefits of monorepo locally and all of the benefits of manyrepo globally, and unless your project hits the ~200+ developer mark maintaining things will be stay tractable.

[radicality]

* Why is ownership management a mess in a monorepo? Can just decide “this team owns this folder hierarchy” etc. * ‘Contain blast radius of changes’ - is that actually difficult? Isn’t there tooling that figures out what changed and what dependencies need rebuilding? (eg Facebook buck)

For context, I was for a very long time at FB so am definitely used to the monorepo way, and recently switched to place which uses github + many repos, and it feels so much worse.

Honest question - how do you actually effectively share code between many repos? Example: How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? It should be a compile/buildtime error for the other project, but how does that work if everything is in its own little repository?

[dragonwriter]

> Honest question - how do you actually effectively share code between many repos?

One way is: Each repo is a responsibility boundary and single source of truth, you use code from other repos the same as any other external dependency.

> How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it?

Changing an API breaks projects using it; you either do versioned APIs and/or coordinate changes with downstream consumers, the same as you would with an API with external customers.

(Another way is “downstream projects checkout their dependencies and build against them as a routine part of their process.“)

[sagarm]

> Changing an API breaks projects using it; you either do versioned APIs and/or coordinate changes with downstream consumers, the same as you would with an API with external customers.

This is a ton of work relative to building/running the tests for all your reverse dependencies and fixing the call sites for them (up to a certain level of scale, of course).

> (Another way is “downstream projects checkout their dependencies and build against them as a routine part of their process.“)

This happens automatically in a monorepo. Any breakages are revealed as soon as upstream makes the change.

[dragonwriter]

> This is a ton of work relative to building/running the tests for all your reverse dependencies and fixing the call sites for them

If someone outside of the team responsible for the functionality at issue can fix all the fall sites for a breaking API change without adversely effecting the domain function of the component making the calls, it's a pretty good sign that there wasn't actually a need for a breaking API change in the first place, and a process change that makes unnecessary API churn more expensive is probably beneficial.

[joshuamorton]

Amusingly I agree with your conclusion that we should discourage breaking API changes, but conclude that monorepos are therefore superior.

Making the team causing the change have to take on the work of migrating clients means that they will make the change as small as possible, and undertake work to to minimize the scope of the change (and make it as backwards compatible as possible).

The alternative is a team publishing a v2.0 API with many incompatibilities and throwing it over the wall, without care that anyone else uses it, and teams eventually needing to migrate to a wholly new API for the one new feature they care about. Or in other words, someone is going to build a compatibility layer, it's more efficient for the owning team to do that, and monorepos encourage that kind of approach.

[giaour]

There are some cases where you have to make a breaking change, and in those cases, it is helpful to be able to assess the fallout without a ton of manual effort. E.g., sometimes a security hole can only be remedied by restricting certain inputs (like with Heartbleed), and that's technically a breaking change.

Monorepos let you make that assessment within a certain scope, as does Amazon's internal build system. It's not a feature you want exercised regularly, but it sure is helpful when you need it.

[giaour]

> Honest question - how do you actually effectively share code between many repos?

Locally, you can use an Amazon-internal tool to check out multiple repos and make changes to all of them. The tooling calls this a "workspace," but it feels very much like working in a monorepo since building and testing can happen at the workspace level.

> How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it?

In terms of change management, Amazon dependency graphs are managed as "version sets." Changes have to be built into a given version set, and that build will also rebuild any package in the version set that consumes the repository whose changes are being built in. (Usually, repositories are configured to build into one of the owning team's version sets on each commit to the primary branch.)

[ignoramous]

> It perplexes me people advocate for this strategy. All I can think is it's another one of those cargo-cult ideas that everyone is doing because Google did it (So it must be good).

Not sure if it is a generic comment or a comment on TFA:

i) If the latter, I'm compelled to point out that TFA doesn't nearly advocate for monorepos as much as it lists reasons why a few SV companies use it, how they use it, and what they get out of it.

ii) If the former, then this blog post makes for a good read: https://tailscale.com/blog/modules-monoliths-and-microservic...

[jvolkman]

Let's not pretend that Amazon hasn't spent significant effort over decades building and maintaining their own (non-monorepo) build systems and tooling.