| > Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready to merge. I'm starting to think most CI problems are just people not looking at the problem the right way. Here is the problem re-worded: - When a PR has a green light and someone hits 'merge', it locks anything else from being to merge to master, and you merge your PR. When it finishes merging and deploying, now all the other PRs waiting have to rebuild themselves to see if they will merge with this new state of master. So 100s of PRs are rebuilding every time you merge one PR, and there's constant CI churn. Here is why that problem exists: - The system was designed for 1000 developers to all be writing to the same code base. Here is how you solve that: - Don't let 1000 developers all write to the same code base. Break the code down into discrete components that different small teams manage. The only bottleneck for that code base is that small team. This small team is often called the two-pizza team, and their discrete components are often called microservices. |