[ben509]

Optimizing for this is a perfect task to throw at a simple market. Especially because actually reworking the software to take advantage of resources at different times is often going to require a decent amount of work by engineers.

One way to do it would be assign various jobs a value, (which could be dynamic e.g. it might get more important as information becomes stale) and have them bid on compute power. You could make the value virtual.

Or you could use real money. This is the premise behind EC2's spot instances. So when power is abundant, your prices drop and the relevant jobs kick off.

Using real market prices makes sense especially if you're renting out computing power, most customers will be happy to adjust workloads to save money.

Even if it's entirely internal, it's good to have a facility to "optimize for cost" and then report the savings. That's helpful to get the engineering resources devoted towards it, because "I saved $X" is a great bullet point to put in anyone's promotion packet or to base a bonus on.

[vladd]

> One way to do it would be assign various jobs a value. Or you could use real money.

It's not the value of the outcome for that job that you're interested, but rather its sensitivity to a delayed latency in executing it.

For example, preemptively converting Youtube videos to a lower resolution with optimum compression to avoid having to do it in real-time (when video is played) at a crappy compression (to be fast), is valuable for sure. It's just that it can be postponed for 24 hours without real impact. Executing a search for a single user is less valuable in terms of overall impact but much more latency-sensitive.

(you can think of value and latency-sensitive in terms of two dimensions that are independent between them.)

This idea helps save the planet for sure, but it requires cloud-providers to build APIs that enable devs to switch from the "here's the SSH to the server, do what you want with it" to a model where it's the devs that say instead "here's a lambda function and its desired latency execution, please schedule to run it for me and let me know when the result is ready" ( https://en.wikipedia.org/wiki/Inversion_of_control )

Google was able to do that because it owns a large part of the jobs executed in their datacenters. Hence they could build this adaptive scheduling for their own jobs quickly without necessarily passing through a cloud-based API that inverts the control of job scheduling.

[mtrovo]

I think this is much more useful to a cloud provider than to a customer.

As a customer I think you could configure somethings like you said using spot instances on AWS but that’s it, you’re going to save some small amount of dollars in a year but if you account for the engineering hours needed to set this up maybe is not really worth it.

As a cloud provider you could juggle your client between datacenters depending on the load and price of energy over there. A flat rate for a cloud region means that there’s an opportunity for arbitration between datacenters that could be an increase of thousands in profits on their side.

[vladd]

If spot instance price is the only communication channel between you and the cloud provider to achieve this, it's hard to do a good job at it. For example, if the spot price was $0.60 one hour ago, and $0.55 right now, and you still have 13 hours of latency left for your job's execution, should you start triggering it or not? (how do you figure out your bid level?) You could have statistical data to have an intuition what's the lowest price that's been hit historically on similar days of the week in the past etc, but it's inexact and overly complicated.

If the cloud provider becomes aware of the remaining latency you have at your disposal for the job's execution, they can do a much better job. They would be able to look across the entire job execution queue in that datacenter, each job having a specific remaining duration for its execution, they would know the predicted pattern of carbon/solar/wind split for the next hours, and the implementation of the system would sit just on the cloud provider side (making the life of the customers easier and simple).

Of course, in the end, the benefit is towards our planet, but this is much more likely to succeed if the proper cloud API exists and the implementation initiatives are properly aligned to avoid redundant implementations on the customer side.

[tylerl]

Thing is, Google requires an absolutely stupid amount of computing resources for running their core business. YouTube transcoding is a great example and a big one for sure, but I bet they have even bigger ones in there somewhere. I have no real data to base this on (and I'm sure nobody does), but I'd bet 5:1 odds that if Google were an AWS customer, they'd be bigger than all the others combined.

So in that case, optimizing for a single customer makes perfect sense if it's the right customer.

[why_only_15]

Something sort of similar exists for iOS. BGTaskScheduler (https://developer.apple.com/documentation/backgroundtasks/bg...) is the public interface, but the internal interface allows for more subtle calculations about how long you can wait until this job is complete and what the optimal physical condition of the phone is (e.g. battery charging, user not using phone, wireless network connection). One of the important features is that when you run a job you have to check in fairly often (every second or so) to see if your job can still run, and if it can't, stop the job. This stops your two hour ML training session from burning battery for a while if the user wakes up at 2am and decides to go on a walk.

[Seirdy]

On a smaller scale, you could do what Low Tech Magazine [0] does and actually have downtime when sunlight is low. Since this doesn't happen too often and users can just save articles (with RSS, email newsletters, etc.), websites like this can just be powered by a single computer, solar cell, and small battery in the owner's Barcelona apartment. Thanks to small static pages and tiny dithered images, the site is almost always up.

The future doesn't always need to be as "webscale" as Google; sometimes, scaling down is the smart thing to do. The minimal approach of LTM is the technology equivalent of riding a bicycle (or electric velomobile [1]) to work instead of driving.

[0]: https://solar.lowtechmagazine.com

[1]: https://solar.lowtechmagazine.com/2012/10/electric-velomobil...

[icebraining]

Actually, I'd say LTM's is the car, and Google's approach is public transport.

LTM's approach required producing, assembling and shipping a full 2GHz/1GB computer, plus PV, PV-controller and router, all to serve a single site. And it's even turned off some of the time!

Google, on the other hand, is more like a fleet of trains; sure, each one is a honkin' beast, but it also transports thousands of passengers/sites at once, possibly millions in its lifetime.

The bicycle analogy doesn't really work, because a bicycle is just a performance attachment to the real vehicle: the human.

[heavenlyblue]

This is actually great: imagine a CDN in various geographical locations all of which work off sunlight in their own time zone (and turn off with no sunlight).

This way you can have 24/7 fully green content delivery to consumers.

Although that being said we could just try to cover the earth with as many generators everywhere and then fully connect the grids.

[toomuchtodo]

> Although that being said we could just try to cover the earth with as many generators everywhere and then fully connect the grids.

This is the easier route: you do this by having as many businesses as possible purchase renewables PPAs, where they're specifically contracting for renewable energy.

https://www.utilitydive.com/news/as-corporate-renewable-buyi...

[samatman]

I respect lowtechmagazine's experiment: it's designed to make you think, and it succeeds.

But CPUs have an incredibly high embodied energy, we should aim for full utilization of servers, regardless of the source of that energy.

If it's three CPUs with carbon-neutral electricity, or one CPU with electricity from coal, the latter is the responsible choice.

[fuzzybear3965]

Did you mean the reverse? I.e. "If it's one CPU with carbon-neutral electricity, or three CPUs with electricity from coal, the latter is the responsible choice."

If no, then why. I can explain my confusion if needed.

[samatman]

I don't mean the reverse, I mean that running one CPU 24 hours a day instead of 3 CPUs 8 hours a day is better, even if the 3 CPUs have carbon-neutral energy and the 1 CPU does not.

To be fair, these are artificial choices, and that's not even what lowtech was doing (they have a battery). I was responding to the post about the CDNs that stop working at night.

[qmarchi]

I mean, technically, that's what Google's solution ends up doing.

Their DCs are globally distributed, so their timings on when they'd actually be doing the heavy lifting is shuffled around.

[skybrian]

It's good to an extent, but if optimization for cost gets too intense then it will seek out the flaws in market rules. This will be true whether machines or humans are doing the optimizing.

I guess it's okay as long as the people making the rules have good monitoring and are watching out for weird exploits and fixing them. The flexibility to change the rules tends to be more common internally than externally where customers want more guarantees.

As we've seen, there also needs to be a balance between cost-optimization and preparedness. If the wind patterns don't match the prediction then you need to be ready for that.

Also, as we've seen with cryptocurrency, real money attracts theft. A human-adjusted credit system is better. In the real world, this looks like support having the discretion to forgive big bills. But to do that they need to know their customers. It's hard to automate.

[uoaei]

Tangent: this discussion is an interesting microcosm of the liberatarian/social-democratic dichotomy of economic theory. GP says a market will take care of itself, parent says not without significant regulation or else the perverse incentives will eventually be exposed and exploited.

[skybrian]

I think it's more a continuum than a dichotomy, since there can be more or less regulation. A regulated market is usually considered to be capitalism. For example, the US stock market is regulated by the SEC. But it does get nebulous as you increase the scope and goals of regulation.

[smadge]

Borg has concepts of quota and priority which function as the internal market you are talking about: Verma, et al. "Large-scale cluster management at Google with Borg".

[johnb]

Tangentially related but in Australia we run a household utility company that operates of that same assumption: https://www.amberelectric.com.au/

Our hypothesis is that market signals combined with the right tools (friendly app and home automation) can help households shift demand into less carbon intensive periods.

So far it's working pretty well.

[bo1024]

Yes! Especially when you have multiple data centers participating in different locations. It might be cloudy in one place but not another, so jobs get re-routed accordingly.

[lukev]

I really hoped this is what EC2 spot instances would be, but it doesn't seem to work that way. My spot instances usually get terminated due to "no available capacity" without any major price movement.

It would also be pretty neat to integrate processing power markets with the wholesale energy markets. Energy prices are quite volatile and making load responsive to that would actually be quite helpful to stabilize them.

[pbhjpbhj]

In the UK households used to be able to get economy 7 electricity, which meant cheap electricity at night (I'd guess for 7 hours?).

I've wanted to have realtime pricing like that for a while, it seems to be becoming available again.

I honestly thought that was what the advanced electricity meter roll-out was going to do; but it seems not.

More direct energy cost to service price charged seems like a good thing in general.

[mikepurvis]

TOU pricing is pretty widespread at this point; I have it in a medium-sized city in Canada, and intentionally run laundry (electric dryer) in the evening when power is half the cost of daytime use. With EVs coming online and being set up to charge at night, it would definitely be nice to have a minute by minute spot pricing scheme, though, then you'd basically have a mechanism for using the chargers and other intelligent devices on the consumption side as an intelligent buffer.

It's goofy, but another one is situations where you have a lot of stored heat energy, thinking like pools, hot tubs, hot water heaters, etc— all those things could be activated in response to spot pricing with pretty simple policies (I want a shower of at least X degrees at 7am, I want the hot tub at at least Y degrees by 9pm, etc).

[walshemj]

I think when widespread overnight charging of EV's is a thing cheap evening electricity will be phased out

[mikepurvis]

Even in that world, you'd still want a way to manage the load through the night so that you don't have all the chargers clicking on at 11pm and then when the cars are full have a few hours later, demand craters until morning.

With renewables the incentives for load management become even higher (per this article). The real next step after second-by-second billing would be setting up chargers with backfeeding capabilities/policies, so that you have an arrangement with your employer to charge your car at work on cheap daytime solar power, sell it back into the grid during the evening rush, then charge up again on overnight base load. Most EV batteries are way overspecced for what people need in daily use, so as long as you have a special "charge me to full and stay there" mode you can switch into, there'd be no reason (other than a bit of wear and tear) not to cycle your battery like this.

[hutzlibu]

I think your household electricity prices just need to reflect real market prices. For example there were times in germany this year, with negative power prices, because the wind was blowing strong and all the wind plants generating more than anyone needed.

But for me as a customer, the price was as high as allways.

But if customers could react to that (automatically), you could have all sorts of jobs waiting for it. Bitcoin mining, dryer, charging batteries, freezer full power ... would be good for the grid, too, to balance it.

[gambiting]

With Octopus you can go to an Agile tariff, where you're paying exactly what the market rate is, which means that frequently at night you are being paid for using electricity(the rate per kWh goes into negative). I know a few people who have it with an electric car, and it means that at certain times you are being paid to charge your car. It's crazy. But also it means that at peak times the cost can be as high as 25-30p/kWh

[ajpgrealish]

There are apps that can automate this whole process for you, monitoring the charge your cars needs each night and the dynamic energy prices either on an Economy 7 or Agile tariff and then controls the charging to optimise the price. If you're car needs less charge than your off peak rate time period then it will charge at the lowest carbon times. https://ev.energy

[gambiting]

OHME charging cables and wallboxes do this as well

[blibble]

https://octopus.energy/agile/

[ftlio]

Smart Meters and Advanced Metering Infrastructure (AMI) still have about half of all meters to go in the US. Real time pricing for the customer requires cutting through a few intermediaries designed to keep the grid and utility bills stable, much at the cost of the environment.

Texas energy markets are where all the fun on this front is really happening.