[zhivota]

I don't see why 3 years is the right number there. I'm using a 10 year old model GPU to generate tokens locally, and given the bottlenecks, a commercial model focused on RAM and transfer speeds should have a longer depreciation curve than 3 years.

[Gigachad]

Because these GPUs in data centers chew power and take up space. If in 3 years there is a new model that processes far more tokens with the same power and time the economics quickly say the hardware is cheaper to replace than to continue running.

As a hobbiest at home the numbers are different and you can afford to do something inefficient.

[zhivota]

I still challenge it. After 3 years an H100 has not depreciated to zero, not even close. Even if you want to replace it, it has considerable residual value.

[wqaatwt]

For some time capital costs seem to be dwarfing the operating costs (or at least energy costs) when it comes to GPUs. Of course that will likely change but it might take quite a while (unless the AI bubble violently pops)