|
|
|
|
|
|
by wizee
338 days ago
|
|
|
You should use flash attention with KV cache quantization. I routinely use Qwen 3 14B with the full 128k context and it fits in under 24 GB VRAM. On my Pixel 8, I've successfully used Qwen 3 4B with 8K context (again with flash attention and KV cache quantization). |
|
|
How many tokens/s? I can't imagine that this would run in any practical way.