[simonw]

Are you talking about the Hugging Face Python libraries, the Hugging Face hosted inference APIs, the Hugging Face web interfaces, the Hugging Face iPhone app, Hugging Face Spaces (hosted Docker environments with GPU access) or something else?

[p1esk]

I updated my comment above: I’m using HF transformers repo, which gets models from HF hub.

[simonw]

Do you have an NVIDIA GPU? I have not had much luck with the transformers library on a Mac.

[p1esk]

Of course. I thought Nvidia GPUs are pretty much a must have to play with DL models.

[objektif]

Well being able to run these models on CPU was pretty much the revolutionary part of llama.cpp.

[p1esk]

I can run them on CPU - HF uses plain Pytorch code - fully supported on CPU.

[tmostak]

But it's likely to be much slower than what you'd get with a backend like llama.cpp on CPU (particularly if you're running on a Mac, but I think on Linux as well), as well as not supporting features like CPU offloading.

[wkat4242]

Ollama supports many radeons now. And I guess llama.cpp does too, after all it's what ollama uses as backend.

[p1esk]

PyTorch (the underlying framework of HF) supports AMD as well, though I haven’t tried it.