[basilgohar]

I can run 7B models with Q4 quantization on a 7000 series AMD APU without GPU acceleration quite acceptably fast. This is with DDR5600 RAM which is the current roadblock for performance.

Larger models work but slow down. I do have 64GB of RAM but I think 32 could work. 16GB is pushing is, but should be possible if you don't have anything else open.

Memory requirements depend on numerous factors. 2GB VRAM is not enough for most GenAI stuff today.