[__strisk]

Is the practice of shoving all disparate pieces of proprietary software (or individual projects) in the same repo a common occurrence? I have found that pulling unrelated changes just so that I can push my changes is an inconvenience. Furthermore, tracking the history of a particular project is confounded due to unrelated commits. I am sure that their vcs (piper?) makes this a feasible task, but for git, it seems like it would suck.

The article posted by kyrra, mentions this.

Given the value gained from the existing tools Google has built and the many advantages of the monolithic codebase structure, it is clear that moving to more and smaller repositories would not make sense for Google's main repository. The alternative of moving to Git or any other DVCS that would require repository splitting is not compelling for Google.

It seems like they have just too much invested in this "shove it in the same repo" style. Or is this the more appropriate way to do things in a large organization?

[throwawayGogEng]

Coming from companies that use reasonably-sized git repos, I absolutely hated Google's VCS.

Here's some of my painpoints with it:

* No branches. If you want to make a temporary code branch, you create a CL (Google's version of a pull request), but never submit it. This means nobody else can collaborate on it with you, and it must be manually updated to HEAD.

* No CL collaboration. Unlike Git branches, CLs can only contain changes from one user.

* No stable branch. Since everything is essentially on one long branch, it's a real hassle when a project is broken at HEAD. Sure, integration tests should ideally prevent this. In practice, HEAD is often broken. Teams have created bash scripts and mailing lists to determine 'stable' old versions that can be checked out for development.

* Single versions of libraries. Any library that is used is also checked into the VCS. However, only one version of the library can exist in the codebase, which is rarely updated. However, there are exceptions to this.

At one point, Sergey mentioned bringing Google "up to industry standards" regarding VCS's. However, that would be a monumental task and I doubt it will happen.

[mikekchar]

It's interesting. This is exactly how it's worked at every large company I've been at. You have these enormous source trees that are essentially unbranchable. Then you have load build teams that build the latest checked in version. When you want to do anything that uses a component you aren't working on, you grab the built version and use it locally. They you hope like crazy that any changes you make don't break any downstream users (who are integrating with yesterday's build). If the build breaks (which happens frequently), then people use builds from 1, 2, or 3 days ago, leading to inevitable "integration hell".

Although I seriously doubt that Google has this problem, one of the biggest drawbacks of the scheme is that nobody knows how to build anything -- not even their own project. If you are coordinating with another project, then you're always having to wait days for the load build to finish so you can get their changes. If the build breaks for an unrelated reason, you can lose a whole week of development to screwing around.

When working in that kind of environment, I tend to maintain my own VCS, then squash my commits when integrating with the main VCS. I also do all my own load build. Everywhere I've worked, I've been heavily criticised (usually by management) for doing this, but productivity has been on my side, so people reluctantly accept it. I often find it strange how so many people prefer doing it the other way...

[throwawayGogEng]

> If the build breaks for an unrelated reason, you can lose a whole week of development to screwing around.

This happens often on some projects at Google (although 2 days is the longest I've seen it broken). Others have told me they use this time for documenting code and writing design-documents.

> When working in that kind of environment, I tend to maintain my own VCS, then squash my commits when integrating with the main VCS.

Google actually has a tool that allows developers to use Git on their local machine, which is squashed into a CL when pushed. However, some projects are too reliant on the old system for this to work.

[jmillikin]

Everything you posted is wrong, which makes me believe you know everything you posted is wrong. Nobody writes things that inaccurate by accident.

For the benefit of YC, here are corrections:

  > No branches. If you want to make a temporary code branch,
  > you create a CL (Google's version of a pull request), but
  > never submit it. This means nobody else can collaborate on
  > it with you, and it must be manually updated to HEAD.

Piper has branches, which can be committed to as normal by any number of engineers. It's common to have both branches for developing certain features ("dev branches"), and branches pinned to a stable base version with cherry-picked bug fixes ("release branches").

  > No CL collaboration. Unlike Git branches, CLs can only
  > contain changes from one user.

CLs are equivalent to Git commits. Collaboration is expected to occur via a series of CLs, just like Git-based projects have large changes made via a series of smaller commits.

  > No stable branch. Since everything is essentially on one
  > long branch, it's a real hassle when a project is broken
  > at HEAD. Sure, integration tests should ideally prevent
  > this. In practice, HEAD is often broken. Teams have
  > created bash scripts and mailing lists to determine
  > 'stable' old versions that can be checked out for
  > development.

There is a global testing system for the entire repository, which is used to decide whether a particular project's tests pass. Commits on which all relevant tests pass are the branch point for releases. This is similar to the Linux kernel's dev model, where stable releases are cut at known-healthy points in an evolving codebase.

Important libraries define more rigorous releases, similar to Git labels, which are updated automatically every day or two. These both reduce the amount of tests that need to run, and reduce chances of errors in low-level code affecting many teams.

  > Single versions of libraries. Any library that is used is
  > also checked into the VCS. However, only one version of
  > the library can exist in the codebase, which is rarely
  > updated. However, there are exceptions to this.

Many third-party open-source libraries have multiple versions, and new upstream releases are added when there's either a security/bug fix, or someone wants a new feature.

Three of the four languages most often used at Google (Python, C++, Go) do not allow multiple versions of a library to be linked into a single process due to symbol conflicts. This is a limitation of those languages, not of the Google repository, and they affect any company that allows use of third-party code. The standard recommendation at Google is to avoid dependency hell by sharding large binaries and using RPCs to communicate. This development model has many advantages that have been documented elsewhere.

[throwawayGogEng]

Thanks for the info!

> Piper has branches

I've worked on two teams so far, neither of which used the branches you described. They opted to either flag off features or keep long-running CLs.

However, I'll try to learn more about Piper branches. I doubt my team will make large workflow shifts, but it would still be good to understand!

> CLs are equivalent to Git commits

I'd argue that CLs are not equivalent to Git commits, the equivalent would be a CL snapshot. Best practices in git are to have frequent, small commits. CLs tend to be much larger, and the review process means that having small CLs would greatly slow workflow.

> There is a global testing system for the entire repository, which is used to decide whether a particular project's tests pass. Commits on which all relevant tests pass are the branch point for releases.

Yes, but this doesn't help with development. When HEAD is broken, a developer has to chose between developing on an outdated codebase, or developing on a broken codebase.

> Many third-party open-source libraries have multiple versions, and new upstream releases are added when there's either a security/bug fix, or someone wants a new feature.

Popular libraries may be updated more often, however other libraries don't have many resources dedicated to them. After a brief correspondence with the team that manages third-party libraries, I decided it would be easier to implement the feature myself instead of following whatever process was required to update the library. And no, I wasn't trying to use 2 versions of the same library.

Despite your assertion, I'm not trying to write anything incorrect, and I appreciate your response.

[JeremyBanks]

While Piper does technically have SVN-like branches, they're so unwieldy and poorly supported by lots of tooling (our team could never get TAP and our slightly-unusual testing to work on branches) that they're a much more niche tool than something like Git branches.

I agree with you with respect to CLs/snapshots. I would sometimes try chains of DIFFBASE-linked CLs in a crude emulation of linear Git branches (I want to experiment with taking things different ways, and I want the version control to store/back up my work), which sort-of worked when you're writing the code, but merging could be nasty. But there was also ad-hoc Git hosting available internally. I started using that for my experimenting, squashing into Piper commits when it was ready to share. It wasn't terrible, though still not optimal for collaboration.

[Bahamut]

> Best practices in git are to have frequent, small commits.

I'd qualify that to only on a working branch - it's preferable to squash it all to 1 commit when merging upstream in order to make reverting painless if there is something wrong with the proposed change.

Lots of small commits cause massive pain with git :( .

[jmillikin]

I don't know which team you work on, so this advice might not be appropriate for your situation. Feel free to email me and I'll try to help you get things sorted. I won't make any attempt to link your corp username to these comments. Same for any other Googlers reading this chain.

  > I've worked on two teams so far, neither of which used the
  > branches you described. They opted to either flag off
  > features or keep long-running CLs.
  > However, I'll try to learn more about Piper branches. I
  > doubt my team will make large workflow shifts, but it
  > would still be good to understand!

Putting new features or changed behavior behind a flag is a good process. Encourage your teammates to avoid long-running CLs in favor of submitting flag-guarded code. You should treat every run of [g4 patch] as a warning that the CL is dragging on too long.

Dev branches require a change in workflow, and can be unhelpful if a team has bigger process issues (like yours sounds like). I recommend looking into them and trying out the codelab, but don't start advocating for them just yet.

Release branches are very important. If your team is not currently using them, that needs to be fixed ASAP. Look into Rapid[1], try it out for a small CLI utility. Advocate for all releases to be done via Rapid. Despite the name it can be slower than plain [g4 sync; blaze build] but the extra steps it runs are important and useful.

[1] https://landing.google.com/sre/book/chapters/release-enginee...

  > I'd argue that CLs are not equivalent to Git commits, the
  > equivalent would be a CL snapshot. Best practices in git
  > are to have frequent, small commits. CLs tend to be much
  > larger, and the review process means that having small
  > CLs would greatly slow workflow.

I'm a Go readability reviewer, so I review a lot of CLs written by people in other parts of the company. My firm belief is that CLs should be small and frequent. CLs start to get hard to review at around 300 lines of feature code. If you are regularly reviewing large CLs, push back and ask the authors to split them up. Often these CLs are trying to do too many things at once and you can find a good fracture point to split them into 2-3 parts.

If you are regularly writing large CLs, or long chains of diffbase'd CLs, that's a sign that your codebase may be poorly factored. Take a step back from the tactical level and look at what your CLs are touching. Is the UI's HTML formatting mixed into the business logic? Are you touching the same file over and over? Move things around, use includes, use data embeds. Replace long param lists with data structures. All this standard software engineering advice applies 2x when working with other devs.

  > Yes, but this doesn't help with development. When HEAD is
  > broken, a developer has to chose between developing on an
  > outdated codebase, or developing on a broken codebase.

If HEAD is broken, your first priority should be getting HEAD fixed. Whether that means fixing the code or rolling back to an earlier version, you should not accept a broken HEAD.

After it's fixed, look at why it broke. Why was a CL that broke things allowed to be submitted? Do you have proper presubmit tests? Consider adding your TAP project(s) to METADATA files so Piper will make sure tests pass and binaries build before allowing the submit.

If other teams' changes are breaking your code, add your TAP project(s) to their METADATA or help them improve their own test coverage.

  > After a brief correspondence with the team that manages
  > third-party libraries, I decided it would be easier to
  > implement the feature myself instead of following
  > whatever process was required to update the library.

Third-party code has special rules that might prevent you from doing something reasonable. Ask the team for help with the process. It's a natural developer instinct to write new code instead of trying to update shared dependencies. If you can fight that instinct and get the new dep version imported, it will improve not just your project but the projects of everyone who might use that dep in the future.

[pyb]

I've heard Google's VCS is basically their own reimplementation of Perforce. Are there any major differences ?

[shemnon42]

Other than "works at Google scale?" That not a dig at Perforce, it scales well but not to the absurd scale Google dials it up to. That is the major feature that drives any differences.

[tehlike]

Git5 bridge exists, though it is not supported.

Single version of a library is actually a good decision, makes sub libraries pulling different version of the same code managable and error free.

Not to mention security auditd and ease of maintenance.

Disclaimer: goog emp

[tehlike]

It actually works quite nicely. Most of the google software is built internally. Ensuring everyone is running at head is a blessing (rarely a curse), because when you make a change you immediately see if it breaks something. You can also be sure everything gets all the bug fixes in their new release. This is good for various reasons, including the fact that it makes security audits significantly simpler.

Some google orgs, like android, don't use the existing infra, and they kind of struggle and have to reinvent most wheels, because of that.

Edit: added second paragraph

[msangi]

The company I work for has a monorepo and has a team dedicated to develop lots of tooling around it to make it manageable.

Do you want to check out only the app you're working on? There is a script that grabs it and its dependencies.

Do you want to do continuous integration? We have plugins to our CI server that understand when the app code or one of the dependencies has been updated.

Building always off HEAD is nice and it solved the issues we had with diamond dependencies but I am not completely convinced this is the right approach

[skybrian]

There are advantages and disadvantages. Here's an article about the good parts:

https://medium.freecodecamp.com/how-google-builds-a-web-fram...

[brown9-2]

I am pretty sure that engineers there don't sync the entire repo to their workspace. Look up the talk that introduced Piper and Client in the Cloud (I think it was at the BUILD conference 1-2 years ago).

[milesrout]

No, it's not common. It's something that a couple of big companies have done, and while it might work for them it does NOT work for most people. It's a really, really bad way of doing development.

[arjie]

Why is it so bad? It seems incredibly effective for them and there are very good outcomes:

• no trailing dependencies so you never have to backport fixes, maintain multiple released versions, etc.

• extensive integration testing of libraries since they go live instantly

• forced library developer and library user collaboration

• deployed code is not out of date

What's the problem with only One True Version? All the complaints seem around tooling.

[latch]

Because smaller shops want the benefit of monorepo without paying the cost/discipline it requires. Mono repos without excellent unit tests and integration tests, review processes and various tools, is a disaster. And, in fairness to Google and Facebook, when they talk about the benefits of monorepo, they always mention these costs (it just gets ignored by people: Headline Driven Development)

And I'm sure that the number of companies doing really good integration tests in somewhere in the 1/10000 range (or worse). Every dependency, external provider, internal API, data store, taking into account versioning....it's hard.

I'm a huge believer in automated testing, I've written about it and done it for almost two decades now. My views on it have continuously evolved...and comprehensive and effective integration testing is still a complete mystery to me. The only reason I'd want to work at BigTechCo would be to learn about that specific aspect of software development.

[Chyzwar]

One True Version force you to move whole company in the same pace. Even if you have legacy project you need to keep updating because your dependencies are moving. In the same time if you want to refactor or redesign API you need to wait for all your users and communicate changes since you cannot have beta/alpha release.

I am guessing that in Google people they try to avoid above issue by creating and then depreciating a lot of projects instead having new major release. Older projects will become frozen because too many things depends on them.

This is visible in Google products that take years to change in any way. Gmail/Search are basically the same as long I remember. Given number of engineers in Google it is hard to see any output.

[milesrout]

Well I suppose my statement was a little strong. The Linux kernel is essentially a monorepo: it contains the kernel and many, many dependent projects (drivers/modules). They do what Google claims to do with their monorepo: change all dependent packages at once if they change an internal API.