[indoordin0saur]

Very curious what hardware you're running this on!

[hypfer]

The same 24GB VRAM RTX 4090 I bought to play Cyberpunk 2077 with.

Works perfectly fine in llama.cpp throwing 70+t/s at me with 128k q8 K/V context when using the IQ4_NL quant + MTP at q4 MTP K/V.

Also leaving this here because you might find it useful: https://hypfer.github.io/will-it-fit-llama-cpp/

[indoordin0saur]

Nice! Do you do anything with that compute when you're not actively using it? Is the crypto-mining hobby still worth it? I've also wondered if such expensive hardware can be rented back out to offset cost. Looks like these cards are going for as much as $4k nowadays.

[all2]

There are services where you can hook your card up and rent it out to other users. I don't know what any of them are called, but they do exist.

[dghlsakjg]

Salad.com is one. (I’m unaffiliated, just happened to come across it this week while looking for a cheap option)

[hypfer]

I've paid ~2k€ in 2023. Since I'm usually sitting next to it, I'm only using it when I want to use it. It can get quite loud and warm.

Crypto (to my knowledge at least) moved away from GPU mining. I guess you could maybe rent out GPU compute, but - being in germany - it's not worth the legal hassle. You could of course always commit tax fraud, though I wouldn't recommend that.

[esseph]

> I've also wondered if such expensive hardware can be rented back out to offset cost.

Massive legal liability. Not worth it.

[Rzor]

Can you fix MTP-GEMMA-4-26B-A4B-IT? It says the weights are 0.5 GB in size.

edit: nvm, I'm confusing models.

[cdelsolar]

What did you call me?