[buyucu]

I have unfortunately been disappointed with the llama.cpp/ollama ecosystem of late, and thinking about moving my things to vllm instead.

llama.cpp basically dropped support for multimodal visual models. ollama still does support them, but only a handful. Also ollama still does not support vulkan eventhough llama.cpp had vulkan support for a long long time now.

This has been very sad to watch. I'm more and more convinced that vllm is the way to go, not ollama.

[mistercheph]

Ollama maintainers seem a bit burned out

[jacooper]

But can you run llms that easily with vllm? do you have to fiddle with formats to get it to run?

[buyucu]

I'm still in early stages of exploration, but vllm seems to be compatible with most models on huggingface.