[mooman219]

Google employs a distributed, caching, incremental build system and a distributed test system across the majority of their code base. I worked in Windows Store and I can assure you that most people there don't use CloudBuild and CloudTest, let along know what they are. I would be confident in saying the majority of people at Microsoft are in that boat.

[haberman]

Just to give others an idea of what this is like: I work on the Protocol Buffers team at Google. Almost all software at Google depends on my team's code. If I have a pending change, I can test it against the whole codebase before submitting (this is a "global presubmit"). If something breaks in global presubmit, I can build and run any target in the codebase with my change in a single command, and this will take O(10 minutes) to build from scratch.

This would be like if I worked on the core Windows SDKs and I could routinely test my changes against everything from Microsoft Flight Simulator to the Bing server code before I submit.

[supernovae]

Is the ChromeOS / Android team like this because it really sounds like we're comparing Epeens.. i'd be surprised if android builds typically build world and do so in 10 minutes..

[shaftway]

There are parts of the company that aren't in this ecosystem, usually for legacy reasons. But even if they were you'd find a lot of this stuff to still be shockingly fast.

Because the build server is centralized it can be aggressive about caching intermediate build steps. Incremental builds aren't just incremental for you, but incremental for everybody.

[general_ai]

Dunno how it is now, but years ago it'd take them a _few weeks_ to just propagate commits into the stable branch through a series of elaborate branch integrations, so yeah, you couldn't change something and test it on a whim. Plus build of just windows alone would take overnight, and rebuilding everything to test a Windows change was not logistically, politically, or technically possible.

[asveikau]

> you couldn't change something and test it on a whim

You could. It would just not leave your branch for a while. Around the scheduled merges it would run against the tests of progressively more of the larger organization.

Parts of this actually constituted a good way to prevent being distracted by the bugs of faraway teams. If something reached your branch, where you were working, it was vetted by the tests required to make it into winmain.

The downside was that people got fairly political about what goes into the branch and when, even for small things.

[asveikau]

Forgot to mention and now it's too late to edit: the most common way to do quick tests during development was to only build a few DLLs or .sys files and replace them on a running system. Then your team would have a set of branches that build every night.

[general_ai]

But that's just Windows. What you can get a Google is a full test run over _everything_ that your change affects. This lets you ensure that an obscure change in behavior will not break others, including products built using your library as a remote, transitive dependency. You also get to fix bugs globally. Say you had a really shitty internal API that was causing problems or slowing things down. You get to actually go in there and change that API, and update callsites in one atomic commit. You can also make sure that you're not breaking anyone's build or tests by introducing your change. There are teams at Google whose purpose in life is repo-wide "code cultivation". Finding issues and fixing them globally basically. This just doesn't happen at MS.

[general_ai]

+1. I know some insiders at MS and their build/test/deployment story is universally very crappy. Things barely work, held together by curse words and duct tape.

Googlers like to joke internally that Google looks like a race car from the outside and like Moving Castle from Hayao Miyazaki'a cartoon from the inside, but that's not the case at all. Comparatively speaking it's a race car inside and out, it's just that the insiders don't know how shitty things are elsewhere.

P.S. I heard Bing is different, but I have no visibility into it, so can't comment.

[christophilus]

I worked in the cloud at MS roughly 7 years ago, and Bing was very different from anything else. The MS cloud services were a total mess. I mean, as in hard to overstate how bad. In contrast, Bing was a well-oiled machine. Simple build, simple deploy, simple devops, simple test, very consistent processes across all of their teams.

At the time, Azure was a joke (partly due to the fact that the initial teams were headed up by ex Office devs with no cloud experience, if I remember correctly). But Azure was cannibalizing the Bing team pretty hard. I hear that strategy worked and that Azure is in much more capable hands now.