[q1w2]

I wonder how many people are scrambling to set this up on their startup infra.

6x24GB NVRAM on 6 GPUs linked with NVSwitch is a little pricey, but totally doable.

[arthurcolle]

I got it running using Colab Pro+ (immediately got a V100 40GB VRAM GPU) - the 7B model works with batch size of 8 and a max seq len of 1024

[koheripbal]

Sure, but the real value here is the 65B. Can you have multiple GPUs on colab?

[arthurcolle]

I can't even get the 13B on colab to do inference with a very small sequence length.

[exo-pla-net]

How pricey would you estimate?

[Tepix]

If you want to do it the cheap way by buying used stuff, the most expensive parts are:

- $2000 for a Threadripper 3xx5WX with a socket sWRX8 mainboard

- $5000 for 6x RTX 3090

- $350 for two 1500W PSUs

- $700 for 256GB RAM

You will also need PCIe extenders and perhaps some watercooling. And find a suitable case. The 2-card NVLink bridges are between $100 and $300 each (you nay want 3). All in all i think less than $10k.

[touristtam]

Would rather put that in AWS.

[Tepix]

AWS is probably among the priciest options.

Privacy costs money.