[asabla]

I'm on a 5090 so it's not apples to apples comparison. But I'm getting ~150t/s for the 20B version using ~16000 context size.

[steinvakt2]

And flash attention doesn't work on 5090 yet, right? So currently 4090 is probably faster, or?

[PeterStuer]

I don't think the 4090 has native 4bit support, which will probably have a significant impact.

[diggan]

> And flash attention doesn't work on 5090 yet, right?

Flash attention works with GPT-OSS + llama.cpp (tested on 1d72c8418) and other Blackwell card (RTX Pro 6000) so I think it should work on 5090 as well, it's the same architecture after all.

[modeless]

Cool, what software?

[asabla]

Initial testing has only been done with ollama. Plan on testing out llama.cpp and vllm when there is enough time