[yazaddaruvala]

Having worked at Google and Amazon.

Honestly their systems are almost identical. Amazon just creates a monotonically increasing watermark outside the “repo”. Google uses “the repo” to create the monotonically increasing watermark.

Otherwise, Google calls it “merge into g3” Amazon calls it “merge into live”.

Amazon has the extra vocabulary of VersionSets/Packages/Build files. Google has all the same concepts, but just calls them Dependencies/Folders/Build files.

Amazon’s workflows are “git-like”, Google is migrating to “git-like” workflows (but has a lot of unnecessary vocabulary around getting there - Piper/Fig/Workspace/etc).

I really can’t tell if the specific difference between “mono-repo” or “multi-repo” makes much practical difference to the devs working on either system.

[safog]

There are no presubmits that prevent breaking changes from "going into live". If some shared infra updates are released, the merge from live breaks for multiple individual teams rather than preventing the code from getting submitted in the first place.

[yazaddaruvala]

I don’t agree with your assessment.

“Merging to live” builds and tests all packages that depend on the update.

So for example, building the new JDK to live will build and test all Java packages in previous live, all of them need to pass their package’s tests, only then will the JDK update be “committed into live”.

The only difference is that Google runs all the presubmits / “dry run to live checks” in the CL workflow. Amazon runs them post CL in the “merge VersionSet” workflow.

[redditor98654]

If it is a new major version of the JDK, then no. Because no existing code would have a dependency on that new JDK major version so nothing gets rebuilt against the new JDK build in live and it will be committed.

If it is a new commit of an existing major version, then yes, existing code in live will be rebuilt before the packaged is committed to live.

[zelphirkalt]

With an appropriately configured CI pipeline, submitted / pushed code does not go live anyway, unless all tests and other checks pass. Unless a test case is missing, which can happen in a mono repo just as well, the code is always checked for the defect.

[humanrebar]

It's impossible to test for every kind of regression. Concurrency and performance bugs are notoriously problematic. At the scales of large codebases, you can have very thorough tests, but they need to be reasonably fast and behave the same way every time they run.

[zelphirkalt]

True, but those notoriously hard to find bugs are still notoriously hard to ind in a monorepo as well. I don't see anything in the character of a monorepo, that would somehow enable me to find concurrency bugs in an easier way than in a non-monorepo setting.

[vineyardmike]

One thing I remember from my time at Amazon that didn’t exist at Google is the massive waste of time trying to fix dependencies issues.

Every week our pipeline would get stuck and some poor college grad would spend a few days poking around at Brazil trying to get it to build. Usually took 3 commits to find a working pattern. The easy path was always to pins all indirect dependencies you relied on- but that was brittle and it’d inevitably break until another engineer wiped the whole list of pins out and discovered it built. Then the cycle repeats. I worked on very old services that had years of history. I’ve often discovered that packages had listed dependencies that went unused, but no one spent time pruning them, even when they were the broken dependency.

At Google, I have no memory of ever tinkering with dependency issues outside of library visibility changes.

Amazon pipelines and versionsets and all that are impressive engineering feats, but I think a version-set was a solution to a problem of their own creation.

[dmoy]

Did you work on a team at Google that uses branches? Most teams do not, so there is no "merge into g3".

[yazaddaruvala]

Every single Fig/Piper workspace is a “branch” in a git-like workflow.

It’s then “merged into g3” from that workspace.

[faizshah]

I haven’t worked at google but I think there is one other difference. At amazon teams “merge from live” and have control of their own service’s CD pipeline. They might manually release the merged changes or have full integ test coverage. The Amazon workflow offers more flexibility to teams (whether or not that might be desirable).

Not sure how deployments and CD work at google but I think the picture is different at google for unit tests, integ tests etc. Amazon teams have more control over their own codebase and development practices whereas, based on what I know, google has standardized many parts of their development process.