[krasin]

> Shortages lead to gluts.

On commodity markets. GPUs are essentially a monopoly, so gluts are only really possible with outdated hardware.

My favorite eBay listing of today is a lot of 100 NVIDIA P106-100 6GB GPUs for ~$20 each: https://www.ebay.com/itm/305023042595

[lhl]

Just in case anyone gets tempted, the P106-100's were mining cards and locked to PCIe 1.1, so the bus bandwidth is terrible. Add to that the limited amount of memory (6GB), low memory bandwidth, Pascal (1/64 speed FP16), and that these were all likely being run in extremely shoddy data centers, it's not even worth the power costs to run IMO.

For those looking for the cheapest higher memory solutions, 24GB P40s are available (decent amount of VRAM, 3X the memory bandwidth, but requires server or DIY cooling, same bad FP16) or IMO the best bang/buck for hobbyists/home devs atm, used RTX 3090s are going for about $600-700 each.

(Note: if you're doing training, unless you have very high utilization/already calculated your costs, you will probably be much better off renting cloud GPUs from Vast.ai, RunPod, etc)

[krasin]

> P106-100's were mining cards and locked to PCIe 1.1

This is not exactly accurate. P106-100 were not specifically made for mining, unlike NVIDIA CMD Hx cards (https://www.nvidia.com/en-us/cmp/). P106-100 cards are NOT artificially throttled, but the specs are still very poor compared to modern cards: https://www.techpowerup.com/gpu-specs/p106-100.c2980

It is true that they are not worth the power costs. I merely made a point that gluts of outdated hardware is a thing even when NVIDIA is a monopoly.

[lhl]

While mining may not be what Nvidia made them for, it's what most of them were sold for, so I think the warning on their likely condition still applies.

Beyond that, I'm pretty sure that the Techpowerup DB is wrong on the PCIe bandwidth. The first-hand reports I've seen online says the P106's are limited to PCIe 1.1:

* https://www.videogames.ai/2019/01/18/P106-100-GPU-Machine-Le... and 2022 followup on P106-90 https://www.videogames.ai/2022/11/28/Stable-Diffusion-Mining...

* https://www.reddit.com/r/lowendgaming/comments/135fymv/bough...

* https://linustechtips.com/topic/1026742-some-information-abo...

$20 is a fair price for those that actually need exactly what the P106 provides, but the cheeky flipside argument is that having a near monopoly might make even the e-waste overpriced. :)

[krasin]

Oh, interesting. Thank you for the correction. I trusted TechPowerUp, somehow...

[otabdeveloper4]

Using a full-blown GPU just for neural network inference is crazy inefficient. They should hire some blockchain dudes to build them custom hardware for one tenth of the price.

[otterley]

NPUs already exist and are everywhere (Apple Silicon, AWS Inferentia). AWS is leading at scale in the cloud in this space.

(Disclaimer: I work for AWS, but opinions are my own.)

[otterley]

I failed to include Google TPUs. My apologies, GCP folks!

[fragmede]

Google hired a bunch of chip designers to make TPUs for their ML people, so are highly optimized for this kind of work, which are currently on their 5th gen, and are available on their cloud.

(Disclaimer: I used to work there but not on them.)