[edwinjones]

Sadly, the reason they give is subjectively terrible:

https://x.com/ollama/status/1952783981000446029

No experimental flag option, no "you can use the fork that works fine but we don't have capacity to support this" just a hard "no, we think it's unreliable". I guess they just want you to drop them and use llama.cpp.

[geerlingguy]

Yeah, my conspiracy theory is Nvidia is somehow influencing the decision. If you can do Vulkan with Ollama, it opens up people to using Intel/AMD/other iGPUs and you might not be incentivized to buy an Nvidia GPU.

ROCm support is not wonderful. It's certainly worse for an end user to deal with than Vulkan, which usually 'just works'.

[edwinjones]

I agree. AMD should just go all in on vulkan I think, The ROCm compatibility list is terrible compared to...every modern device and probably some ancient gpus that can be made to work with vulkan as well.

Considering they created mantle, you would think it would be the obvious move too.

[MindSpunk]

Vulkan is Mantle. Vulkan was developed out of the original Mantle API that AMD brought to Khronos. What do you mean "AMD should just go all in on Vulkan"? They've been "all in" on Vulkan from the beginning because they were one of the lead authors of the API.

[dagmx]

Vulkan is a derivative of mantle sure, but it is quite different than what Mantle was.

There was a period in between where AMD basically EOL’d mantle and Vulkan wasn’t even in the works yet.

[edwinjones]

I would say vulkan derives from mantle, mantle development stopped some time ago.

[zozbot234]

iGPUs (and NPUs) are not very useful for LLM inference, they only help somewhat in the prompt pre-processing phase. The CPU has worse bulk compute but far better access to system memory bandwidth, so it wins in token generation where that's the main factor.

My conspiracy theory is that it would help if contributors kept the Vulkan Compute proposed support up to date with new Ollama versions; no maintainer wants to deal with out-of-date pull req's.