[c0g]

Does anyone have experience of putting a huge GPU on something like this and using it for inference? You'd be limited by data feeding over the NVME port, but otherwise you won't be bottlenecked right? Seems like a light weight and cute way to limit non-inference power/weight without having to pay the price of a Jetson board.

[rcarmo]

You just don’t have the bandwidth to do that. Even if you use the M.2 slot you’ll be significantly bottlenecked and would be better off using something else - even an AMD iGPU will work better (https://taoofmac.com/space/blog/2024/04/13/2100)

[c0g]

What’s the bottleneck? Once I’ve got the model and data onto the GPU my only cost is launching CUDA kernels right?

Not sure if that blog post is relevant, but even if it is it shows a 3060 gets //way// faster throughout than the igpu it is testing. I suppose I can test this myself by plugging my 3070 into the NVME on my desktop.

[me_me_me]

I think you mean plugging a mobo into your RTX?

On serious note, since you need big psu to drive serious GPU, why opt to use RPi or any other small form factor?

[c0g]

It’s a mobile platform, so saving 100 w on the CPU would make a difference. That’s the answer I tell myself, the real answer is because it would look hilarious :D