[bizzleDawg]

It seems that it must be a really difficult problem to work out the optimal solution for having spare capacity to allow time/location shifting of workloads to minimize carbon per unit of compute.

This Dell paper[0] suggests that 16% of the carbon over a typical server lifecycle is from the manufacture, so you probably don't want a server sitting there unused for 23 hours per day, since the overall carbon/compute ratio would be worse overall.

The post doesn't mention this metric, but it would be really nice to see something more detailed in time - especially with this overall efficiency of the server/datacentre lifecycle in mind, rather than just energy consumed from use.

[0]: https://i.dell.com/sites/csdocuments/CorpComm_Docs/en/carbon...

[shadowgovt]

Carbon consumed in building a server is sunk cost and would be paid independent of whether the server does any kind of carbon-footprint-aware load shifting.

Assuming the server is "sitting unused for 23 hours a day" is the wrong model for what this work changed. You're assuming the server could be running at 50% duty cycle vs. 100% duty cyle. It isn't; since we're talking the batch load, there's a roughly fixed amount of low-priority work to be done and doubling the amount of CPU active-duty time alotted to doing the work doesn't get the work done faster (the details on that are complicated, but that's the right model for what Google's describing here). One should model the duty cycle as fixed relative to the processor (i.e. "This global datacenter architecture, over the course of its life, will do a fixed N units of computronium work on these batch tasks") and then ask whether that work should be done using coal to power the electrons or wind.

[gbear605]

Suppose I'm building a new datacenter that I want to do some constant amount of work each day. It doesn't matter the time of day. I can either power it with solar power, in which case it will run for 1/Y of the day, or with coal power, in which case it will run 100% of the day. If it only runs for 1/Y of the day, then I will need to buy Y times as many computers in the solar scenario than in the coal scenario.

If Y = 2 and only 16% of the carbon in a typical coal-powered computer's lifetime is from the manufacture, then solar makes sense - solar is 2*16% = 32% of the carbon of coal. But if Y = 10 - so it's running 10% of the time, meaning there need to be 10x as many computers built - and 16% of the carbon is from the manufacture, then solar power is actually worse for the environment than coal power: solar takes 60% more carbon than coal power.

Of course, this is a vastly simplified situation, but it points to the idea that we need to at least consider the carbon cost of manufacturing.

[shadowgovt]

But again, that's the thing. There is only 1/Y work to do in the day; it's batch work. The work in this case is constrained on the input side, not the CPU resources side; building 1 or 2 or 1,000 nodes to do the work won't decrease how expensive it is to do the work (in fact, building more computers than you need will make it cost more!).

... so why does Google build more computers than they need? Keep in mind that at Google scale, they're always and forever building "As many computers as we can possibly afford to" under the assumption that there will always be work for those machines to do. You and I may need to consider cost of manufacturing; Google doesn't. They always have the "Build datacenter infrastructure" cranked to an 11 (more accurately, they are following an N-year plan of construction that is extremely expensive to modify).

That's the breakdown between Google's way of thinking and the way of thinking that you've presented: Google's cost of manufacture is fixed. Those computers will be built, whether or not they're going to also then run green-streamlined batch jobs. The limiting factor on the batch work is only so much work is generated in a day, and the rate the work is completed is already good enough that completing it in half the time yields no marginal value. So may as well complete it using sun instead of coal.

[bizzleDawg]

> Google's cost of manufacture is fixed.

That may well be the way you (and most likely Google) currently look at it, but the argument presented seems to be that everything other than where you run some batch tasks is all fixed.

Assuming everything else is fixed does make the optimisation easier, though surely nearly everything is up for debate if we're actually talking about minimising the overall footprint of a compute task?

Using fossil fuels rather than allowing these fixed costs you mention to sit idle - there is a point where your carbon/CPU cycle would actually go up by not using fossil fuels some of the time (though I am assuming less carbon emitted for a unit of power than manufacture cost).

I appreciate they are ever-expanding and predicting where non-movable workloads are going to need to be run etc etc, and I'm not suggesting there are easy answers.

[shadowgovt]

Is the question on the table whether, all other things being equal, this change would decrease carbon output per unit work completed, or is the question whether Google is greener after this change?

For the latter question, we have insufficient data. Google's datacenter infrastructure is huge and complicated, and if one factors in all the interdependencies and purchased carbon offsets, one needs way more data than this announcement blurb gives out to answer that question. Maybe they have an additional process to determine whether their most carbon-negative datacenters can be switched off during peak coal-use and this work unblocked them from enabling that feature? We don't know.

For the former question, yes.

[hinkley]

If I have 2 hours of work to do a day, I could sell 90% of my servers, the racks, the networking hardware, one of the air conditioning units, backup generators, and battery backups, lay off some of my IT staff, and just let the thing run 23 hours a day.

What people are trying to tell you is you can’t make having a glut of equipment work out for any reason, let alone environmental. It costs you a dozen times over to have most of you hardware not doing anything at all most of the day. You are overprovisioned.

You are so overprovisioned in fact that someone will steal your promotion by pointing this out to your boss, which is also a pretty big cost.

[shadowgovt]

> What people are trying to tell you is you can’t make having a glut of equipment work out for any reason, let alone environmental.

Google has a glut of equipment for two reasons: variable loads and experimental.

The batch loads are predictable and constrained by input rate. What the coal ends up used for is spikes in demand for compute power (i.e. "Michael Jackson died and now everyone needs to watch Thriller right now") and ceiling to develop and experiment with new projects.

This optimization means that while, yes, the fossil fuel resources would be feeding those use cases, it's a net gain because those use cases are less common than the predictable batch-job work.

> You are so overprovisioned in fact that someone will steal your promotion by pointing this out to your boss, which is also a pretty big cost.

Remember the context. Nobody ever got fired at Google for adding more compute power to the behemoth, because it's all fungible and will all be used eventually. if you're trying to argue "Google is a net polluter because they continue to build machines in a fashion only bounded by land and hardware costs under the assumption more compute power is always better than less" it's an argument you could make but (a) you should factor in the carbon offsets they buy and (b) you're tipping close to arguing "All human activity is pollution; kill the species."

[heavenlyblue]

Doesn’t not buying servers create pressure to not build them in the first place?

[shadowgovt]

I shouldn't have used "sunk cost;" I should have said "fixed cost."

Google builds datacenters to N-year-long plans that are expensive to modify. Whether they're running their batch jobs on solar energy or coal energy, the carbon footprint of the datacenter build plan is not going to change. They want those datacenters anyway (mostly for the non-batch work and whatever the next big thing is that hasn't been invented yet but can only be done by putting a beach's worth of thinking sand on the problem).

[webdva]

It is ultimately, in a certain sense, a mathematical optimization problem to determine the optimal configuration of the entire infrastucture of additional power sources. Perhaps like finding the optimal location position set of cell phone towers, perhaps using k-means clustering. Furthermore, additional issues must be resolved like legal regulations compliance—the decision maker or engineering agent has preferences or desires to satisfy.

[adrianmonk]

> for having spare capacity to allow time/location shifting

Part of that calculation should be the amount of compute capacity headroom you'd choose to have anyway even if you didn't care about carbon.

Compute demands can vary from one day to the next. Maybe tomorrow people uploaded 3 times as many YouTube videos as they did today. Maybe load varies based on day of the week or day of the month. To some extent, you can smooth that out by delaying jobs, but there are practical limits.

You also want some spare capacity just for safety. Efficient utilization is important, but things like performance regressions or spikes in demand can happen.

[ip26]

Two possibly examples-

- Spawn nightly regressions when wind power starts to pick up, instead of at some arbitrary wall clock time

- Dispatch compute-heavy jobs during low energy cost times; dispatch IO-heavy or memory-limited jobs during high cost times.