[iamhungry]

We don't even collect statistics about that for ranking changes.

Who's "we"? Are you Larry Page to speak with such confidence? Nope, just a lowly engineer being observed by hundreds of Stanford and ivy League MBAs as your changes impact Ad Clicks, the golden eggs.

[moultano]

I know you're basically just trolling, but I'll reply seriously anyways.

I'm one of the people who makes changes to Google's rankings. As a result, for any change I want to make, I have to collect the statistics to justify it. This is done with the help of an analyst who has a different reporting chain from the ranking engineers in order to ensure that they remain unbiased. These statistics include things like what results people click on, how often people hit "next page," how humans rate the results before and after the change, etc. (None of these statistics involve ads or revenue in any way.) Once we've collected those statistics, the analyst writes up a report about the change summarizing their findings and pointing out any areas of concern.

This report is then presented at a weekly launch meeting, where the ranking leads review each change both for its metrics and for its complexity, ongoing infrastructure cost, etc. and make a decision about whether to launch it. You can view an example launch meeting here: http://www.youtube.com/watch?v=JtRJXnXgE-A

[nostrademons]

FWIW, for most engineers the entire Search org chart, all the way up to Larry, is engineers. Larry was an engineer, Alan Eustace was a hardware engineer, Amit Singhal is an engineer. In my management chain there are two people with an M.S. in computer science (Larry is one of them), zero with MBAs, and everybody else has a Ph.D in CS.

If your mental model of Google is a bunch of MBAs who tell engineers what to do, you're quite incorrect. That's Microsoft.