Hacker News new | ask | show | jobs
by eso_logic 17 days ago
No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K80's 24GB of VRAM :0
1 comments
russianGuy83829 15 days ago
Thank you, looking forward to it.

I just saw this simple patch to enable MTP (potentially 2x performance) on older GPUs (Kepler etc), so maybe it will work for you

https://github.com/ggml-org/llama.cpp/pull/25680

Also, for Qwen, the 4 bit _XL quantization seems to have a good balance of performance to size.

link