[p1esk]

I updated my comment above: I’m using HF transformers repo, which gets models from HF hub.

[simonw]

Do you have an NVIDIA GPU? I have not had much luck with the transformers library on a Mac.

[p1esk]

Of course. I thought Nvidia GPUs are pretty much a must have to play with DL models.

[objektif]

Well being able to run these models on CPU was pretty much the revolutionary part of llama.cpp.

[p1esk]

I can run them on CPU - HF uses plain Pytorch code - fully supported on CPU.

[tmostak]

But it's likely to be much slower than what you'd get with a backend like llama.cpp on CPU (particularly if you're running on a Mac, but I think on Linux as well), as well as not supporting features like CPU offloading.

[p1esk]

Are there benchmarks? 2x speed up would not be enough for me to return to c++ hell, but 5x might be, in some circumstances.

[wkat4242]

Ollama supports many radeons now. And I guess llama.cpp does too, after all it's what ollama uses as backend.

[p1esk]

PyTorch (the underlying framework of HF) supports AMD as well, though I haven’t tried it.