[csmpltn]

Tracking "Energy" for what is otherwise a compressed version of the entire internet at your disposal seems so green-washed and disingenuous. Whatever "Energy" consumption those models have - it's peanuts compared to the alternatives.

[hdkrgr]

I had the chance to talk to a staffer of one of the MEPs leading the political negotiations in EU parlament committee a few weeks ago. His take was that the pro-tech/pro-business parties conceded the 'AI users must track their energy use'-point to the Greens in the latest draft (which is the parliament's counterproposal to earlier drafts by the commission --think EU executive-- and council --think governments of the member states--) because it's so unrealistic in practice that it's likely to be stricken out of the law again during the final negotiation round between parliament and council negotiators.

I really hope that'll be the case. FWIW, I believe companies _should_ be required to keep tabs on their (and their supply chain's) emissions, but demanding that this be done at model/system level by data scientists is just ridiculous.

edit: grammar

[londons_explore]

The problem is, as soon as it's a law, there will be some official way to calculate it, penalties for misreporting, perhaps even a professional who must audit the energy use. Etc. Getting that number gets expensive!

Whereas if it is a company voluntarily reporting it, the number would just be number of GPUs * wattage of GPUs / tokens generated past year = energy per token.

[hdkrgr]

I agree and share this concern in principle.

But... have you seen the state of GDPR enforcement? Anyone who made an honest effort is fine. I don't know of any GDPR enforcement action where the indicted company wasn't blatantly and willfully violating or ignoring the law.

FWIW, everything I've seen from regulators and the legislators involved in the nitty-gritty of the act seems to suggest that most of them are really smart people who know what they don't know. They know that AI is quickly evolving and the draft of the law goes out of its way to _not_ be too specific about _how_ to comply. E.g., I would not expect the EU (or national regulators) to bring down 'one right way' to report energy consumption.

The fact that they COULD still bothers me.

[senko]

It's still an useful metric, why not measure and report it? Especially for models that are being deployed on mobile phones or desktop apps. And looking ahead, the future where IoT devices will have a mini-neural-net is not inconceivable.

One of the lessons from the crypto fiasco is that, if unchecked, the energy requirements can baloon to stratospheric heights.

[hdkrgr]

I do think this will be a useful metric, and it seems obvious that the hyperscalers will have a feature helping you keep track of energy use and emissions of the resources you rented. But why demand this on the level of an individual model/product? For these foundation models, I think it's reasonable to assume they will all be trained on hyperscaler-provided gpu-clusters, so there'll likely be an off-the-shelf funcitonality by AWS/Azure/GCP to report this number, but the draft of the EU AI Act also demands tracking energy use for other 'high-risk' AI systems which companies may plausibly train and/or deploy on-prem. Good luck tracking the per-token energy use of your model that's running on some on-prem server on last-gen GPUs.

[Dylan16807]

Especially for a server GPU, looking up watts and multiplying by time per token should give you a pretty good number.

[hdkrgr]

Sure... but maybe the GPU is sitting idle 40% of the time while still consuming 200W. Should I have to break this idle energy consumption down onto actual use (assuming the server/gpu is only used for this one model)? I guess it would make sense, but... WHO should do this and then continually update the model documentation when idle rates or the hardware changes?

[senko]

The organizations that release the models already provide (brag about) their model performance. They could simply include in the same report the info about the energy spent doing the training/finetuning/inference, per X tokens.

This doesn't necessarily measure every use, just "manufacturer's spec", the same you get for eg energy class for house appliances (at least in the EU). Nobody goes around measuring refrigerator power usage, but when you're buying one, you get a rough indication of how "green" (or not) it is.

[hdkrgr]

I agree, that seems reasonable!

I was referring more to the users of such a system (what the AI Act would call a 'deployer'). They may have significantly less expertise but could still be required to track real-time energy use. Of course, simply referring to the 'energy label' by the provider could be a viable solution.

[Dylan16807]

Listing it per server design (with groups) makes sense to me.

It wouldn't make sense to include measured idle time in the energy numbers you'd include in model documentation. Maybe that could go in a monthly report somewhere, but that's a different topic.

[user34283]

„It’s useful, why not“ is not a good basis for regulation.

Although the EU seems to think otherwise time and time again.

[csmpltn]

> "It's still an useful metric, why not measure and report it?"

What is it actually useful for?

[gchadwick]

I believe the concern is everyone wants a foundation model, each one may have a comparatively tiny energy/carbon cost (vs yearly global usage and carbon output) but if there's lots of organisations training them up this all adds up. The energy data can help decide how much of a problem this is to worry about (solved through open models you can fine tune rather than always starting from scratch or maybe some regulations around model licensing that looks like the FRAND terms you see for patents and other IP.

[gmm1990]

Pretty sure a google search is order/s of magnitude cheaper than a chat gpt one. Training costs may not be comparable. I’m not sure.

[jdiez17]

The energy/environmental costs may be small compared to the potential usefulness of foundation models, but it's hard to make that assessment without data.