[j45]

An individual token, and the level of energy it represents (electricity, or relative effectiveness per model) increasingly seems the space of obsfucation.

This space can be increasingly avoided by becoming, and remaining, efficient and effective with prompts.

[dinkleberg]

That is one of the interesting things about Neuralwatt cloud. Their pricing is based on energy rather than tokens (actually they have a token-based alternative, but claim the energy pricing results in 95% cheaper results). I've tried out their subscription offer and it does seem like you get a lot more usage even on the cheap plan. However since the energy metering is pretty much unique to them (at least that I've seen), there really isn't anything to compare it against, so hard to tell how accurate it is.

Regardless, it is cool to be able to contextualize the actual spend in terms of physical energy utilization. It even has a little co2 number (though again, kind of a "trust me bro" metric).

[scottcha]

Hi, I'm co-founder of Neuralwatt. While there aren't other providers selling by energy we also produce the same tokens stats you get at others (input, output, cached) and you can compare using those. The CO2 number is really just multiplying the energy by the CO2-intensity at the time/place we served the request. So everything is actually measured and can be derived externally.

The main observation when you compare to token prices our input/cached energy is much lower than the equivalent token prices while the output energy is generally higher. One of the reasons this is a bit cheaper, especially for agentic systems, is that in an fully built/cached session your input & cached to output ratio is really massive input to very small output (multiple orders of magnitude usually). The other thing we do is we actively work to optimize the system around tokens/joule with the goal of making the most energy-optimal system.

Happy to answer any other questions.

[j45]

I think I saw something like that, and it made me realize the watt isn't the only measurement. The watt can be spent effectively, or ineffectively, thus obsfucating mileage per watt.

How we drive AI will cause mileage variances, but over time the improved practices can measurably change.