[Muskyinhere]

Because if they could just do that and it would rival what NVidia has, they would just do it.

But obvoiusly they don't.

And for reasons: NVidia has worked on CUDA for ages, do you believe they just replace this whole thing in no time?

[Wytwwww]

Does CUDA even matter than much for LLMs? Especially inference? I don't think software would be the limiting factor for this hypothetical GPU. Afterall it would be competing with Apple's M chips not with the 4090 or Nvidia's enterprise GPUs.

[Der_Einzige]

It's the only thing that matters. Folks act like AMD support is there because suddenly you can run the most basic LLM workload. Try doing anything actually interesting (i.e, try running anything cool in the mechanistic interoperability or representation/attention engineer world) with AMD and suddenly everything broken, nothing works, and you have to spend millions worth of AI engineer developer time to try to salvage a working solution.

Or you can just buy Nvidia.

[treprinum]

llama.cpp and its derivatives say yes.

[m00x]

This is the most script kiddy comment I've seen in a while.

llama.cpp is just inference, not training, and the CUDA backend is still the fastest one by far. No one is even close to matching CUDA on either training or inference. The closest is AMD with ROCm, but there's likely a decade of work to be done to be competitive.

[yumraj]

Yes, and inference is a huge market in itself and potentially larger than training (gut feeling haven’t run numbers)

Keep NVIDIA for training and Intel/AMD/Cerebras/… for interference.

[latchkey]

The funny thing about Cerebras is that it doesn't scale well at all for inference and if you talk to them in person, they are currently making all their money on training workloads.

[m00x]

Inference is still a lot faster on CUDA than on CPU. It's fine if you run it at home or on your laptop for privacy, but if you're serving those models at any scale, you're going to be using GPUs with CUDA.

Inference is also a much smaller market right now, but will likely be overtaken later as we have more people using the models than competing to train the best one.

[Muskyinhere]

NVidia Blackwell is not just a GPU. Its a Rack with a interconnect through a custom Nvidia based Network.

And it needs liquid cooling.

You don't just plugin intel cards 'out of the box'.

[superkuh]

You're not wrong, but technically llama.cpp does have training (both raw model and fine tuning). And it's been around for a long time. Back around the ggml->gguf switch I used llama.cpp to train a tiny 0.9B llama 1 through the early fast parts of the loss reduction on 3GB of IRC logs with 64 tokens of context over about a month. It eventually produced some gpt2-like IRC lines within it's very short context.

Would anyone choose llama.cpp's training tools to do serious work? No. Do they exist and work, yes.

[treprinum]

Inference on very large LLMs where model + backprop exceed 48GB is already way faster on a 128GB MacBook than on NVidia unless you have one of those monstrous Hx00s with lots of RAM which most devs don't.

[Muskyinhere]

No one is running LLMs on consumer NVidia GPUs or apple MacBooks.

A dev, if they want to run local models, probably run something which just fits on a proper GPU. For everything else, everyone uses an API key from whatever because its fundamentaly faster.

IF a affordable intel GPU would be relevant faster for inferencing, is not clear at all.

A 4090 is at least double the speed of Apples GPU.

[treprinum]

4090 is 5x faster than M3 Max 128GB according to my tests but it can't even inference LLaMA-30B. The moment you hit that memory limit the inference is suddenly 30x slower than M3 Max. So a basic GPU with 128GB RAM would trash 4090 on those larger LLMs.

[skirmish]

Quantized 30B models should run in 24GB VRAM. A quick search found people doing that with good speed: [1]

    I have a 4090, PCIe 3x16, DDR4 RAM.
    
    oobabooga/text-generation-webui
    using exllama
    I can load 30B 4bit GPTQ models and use full 2048 context
    I get 30-40 tokens/s

[1] https://old.reddit.com/r/LocalLLaMA/comments/14gdsxe/optimal...

[m00x]

Do you have the code for that test?

[m00x]

Because the CPU has to load the model in parts for every cycle so you're spending a lot of time on IO and it offsets processing.

You're talking about completely different things here.

It's fine if you're doing a few requests at home, but if you're actually serving AI models, CUDA is the only reasonable choice other than ASICs.

[treprinum]

My comment was about Intel having a starter project, getting enthusiastic response from devs, network effects and iterate from there. They need a way to threaten Nvidia and just focusing on what they can't do won't bring them there. There is one route where they can disturb Nvidia's high end over time and that's a cheap basic GPU with lots of RAM. Like Ryzen 1st gen whose single core performance was two generations behind Intel trashed Intel by providing 2x as many cores for cheap.

[m00x]

It would be a good idea to start with some basic understanding of GPU, and realizing why this can't easily be done.

[pjmlp]

A fraction of CUDA capabilities.

[treprinum]

Sufficient for LLMs and image/video gen.

[pjmlp]

A fraction of what a GPU is used for.

[m00x]

FLUX.1 D generation is about a minute at 20 steps on a 4080, but takes 35 minutes on the CPU.

[vunderba]

Yep. Any large GenAI image model (beyond SD 1.5) is hideously slow on Mac's irrespective of how much RAM you cram in - whereas I can spit out a 1024x1024 image from Flux.1 Dev model in ~15 seconds on a RTX 4090.

[treprinum]

4080 won't do video due to low RAM. The GPU doesn't have to be as fast there, it can be 5x slower which is still way faster than a CPU. And Intel can iterate from there.

[m00x]

It won't be 5x slower, it would be 20-50x slower if you would implement it as you said.

You can't just "add more ram" to GPUs and have them work the same way. Memory access is completely different than on CPUs.

[Der_Einzige]

Not even close. Llama.cpp isn't even close to a production ready LLM inference engine, and it runs overwhelmingly faster when using CUDA