[fkdjs]

The problem with that is, the unsexy non-customer facing projects that allow for saving/making millions are not recognized, or at least a monetary value cannot be established for them, and become understaffed, which leads to inefficiencies. To get to the big bucks, you really have to look out for #1 and join the sexy groups. Which is partly why you see things like reader die: reader does not a bonus/promotion make. You gotta be churning out new stuff at minimum, hopefully on sexy new products, maintenance of the old stuff is less sexy and doesn't give you the promotion/bonus. Of course you can reimplement a cache server every 6 months, if all else fails.

[tytso]

Actually, in some ways it's easier for those of us who work in the infrastructure groups to demonstrate monetary value. Google has a very large number of machines, and an even larger number of hard drives. Hard drives have fundamentally not changed seek times in over a decade, even as capacity has doubled (up until relatively recently) every 18 months. So more often than not, for many work loads (not just at Google, but across the entire industry) are seek constrained, not capacity constrained. So when Google migrated from ext2 to ext4, it's actually pretty easy to calculate how much money was saved by utilizing disks more efficiently, and the team which spearheaded this _was_ in fact recognized by the company and by the founders.

As far as the need to continue to derive new value to the company, of course! This is true everywhere; the phrase "what have you done for me lately" is one that is not unique to any one company, and would _you_ respect someone who did one great thing many years ago, and then proceeded to rest on his or her laurels?

Of course there are many different ways of adding new value. If you can demonstrate how a new cache server is faster or more scalable, and thus can drive value to the company, that's certainly one way. Or maybe there is new technology, such as faster networking technology or faster flash storage, which means that assumptions made five years ago are no longer true, and that can be a justification to rewrite some part of the infrastructure. But Google is a data-drive company, which means you need to be able to justify why the rewrite is necessary.

[fkdjs]

That is great if your project impact can be measured, but sometimes metrics aren't available for accurate assessment. If sales uses your tool, bringing in millions, that's not so easy to measure, and those projects will languish unless some value can be placed for the project. Good luck with that, or at least coming up with a system that can measure it and having it be accepted. That's a big gap and you don't want to be in that group.

[tytso]

Do you really work for Google? It's possible to find metrics for most things. It might be the number of times SRE's get paged. It might be reducing the amount of time SWE's need to worry about thread safety; so even code cleanup can have metrics. The trick is to think big, and to think at scale. Not how to improve things for SWE's working in one team, but many teams, if not all of Google.

This is why life is sometimes easier for the infrastructure teams; increasing disk or CPU utilization by even a fraction of a percent, when multiplied across a large number of systems, can be a big number.

Now, if your product which is measured as having a small number of users, and worse, that number is decreasing over time (which was the case with Google Reader, as has been publicly disclosed), the problem is not that you don't have metrics, but the metrics aren't telling you want might have necessarily wanted to hear. Down and to the right; that's a different story...

[fkdjs]

Yup, and it's used by everyone, especially execs, and acts as a swiss army knife. Yet the team is mostly leaving in droves because although it's quite useful, it isn't correctly measured and it doesn't bring home the bacon. So the project is sometimes wrapped up and repackaged by a customer-facing team to steal the glory and get bonuses, or teams put it under them and scale back perks/force us to use them vs other teams who want us to cater to them. So silo'ing, everyone wants the goods. I've contributed lots of infrastructure code that customer-facing teams use, gets who gets the large bonuses. Proposals to fix this and measure impact have been brought up and slapped down by managers. I'm leaving the group after wrapping up some stuff. Fortunately I have high perf scores. People make warm comments about how useful the project is, but at the end of the day, it's never going to give me a $200k bonus, so it goes in one ear, out the other.