[oceanplexian]

At least with a GPU that can do power save that's not the case. I have a box with some 3090's in it, each card will idle <50W when it's not doing inference with the weights loaded into VRAM. Only when I ask it to do inference it will spin up and start consuming 300-400W.

[ghthor]

I can confirm this, unless my llm is doing inference nvtop reports idle level wattage.