[ineptech]

Might as well add my own experience since I just set up a local llm this week. I went with a 32GB card made by Intel called Arc B70, which is cheaper than a 3090 and more has ram, at the cost of a slower memory bus. edited to remove something incorrect, thanks diablod3

I went with this because a) the models I wanted to use are a little too big to fit comfortably in 24gb, plus I need room for a few additional small models for autocomplete and speech recognition, and b) I already had a cheap server to use and dual gpus would've required upgrading the mobo and power supply and probably the case as well.

It was definitely a little tricky to set up. The Intel line requires a driver package called "level zero" to support something called SYCL (Intel's version of CUDA basically, AFAICT) that was tricky to get working. I am running llama.cpp in a docker container, which also required some fiddling to get the container to see the card. You also need a kernel from the last few months.

Once I got it working though, the results are very impressive for a $1k investment. Qwen 3.6 35B at q4 quantization takes about 3/4 of the ram and delivers like 88 tokens/sec. So, if you want a decent-sized model for cheap, this is one way to go.

[DiabloD3]

That is incorrect.

They both have GDDR6.

The B70 has 256 bit it bus at a clock speed of 2375mhz (608 GB/s), the 3090 has a 384 bit bus at a clock speed of 2438mhz (936 GB/s).

It isn't slower, it just has less channels, ie, it is less wide.

[ineptech]

Whoops thanks, was going from memory. At any rate, the effect is that it's somewhat slower than the 3090, when using a model small enough to fit entirely in nvram, but can fit models the 3090 can't.

[DiabloD3]

Yep, but the flip side is you can get two B70s for the price of a single 3090 (MSRP, obviously; 3090s used go for about the same as a new B70), and they're true 2 slot, so they can fit on x8/x8 consumer boards fine.

The side effect is Intel has fired their entire Arc team, including the driver team, as well have canceled all Celestial products (only low end combined GPU + IO die products will have Celestial, as its too late in production to change). All future Intel products will have Nvidia graphics tiles.

Llama.cpp is, of course, still trying to better support the existing Intel products, but its kinda hard when they might stop working any day due to driver breakage.

[mstaoru]

"Intel has fired their entire Arc team"? "All future Intel products will have Nvidia graphics tiles"?

Can you point to a source of these wild claims?

[DiabloD3]

HN front page circa September through December of last year. Its part of the $5B investment into Nvidia, and the new CEO, Lip-Bu Tan, happily did whatever Nvidia told him to do.

Nova Lake (Series 4) is already too late in development to switch, so its shipping with Celestial, and Titan Lake (Series 5) after it will be essentially a Nova Lake refresh using a hybrid tile that is Celestial's graphics engine with Druid's media, memory, and bus controllers (since Druid development has been halted).

Serpent Lake (Series 6) will be the first that can ship with Nvidia tiles, which should launch in early 2028.

Unless something changes, there will be no DGPU Celestial products.

Unfortunately, the ink on the contract is already dry, so this is the end of Intel's GPU unit.

[mstaoru]

I see. "My" (double checked) information is:

- Arc discrete (!) gaming (!!) GPUs are canceled,

- consumer gaming (!) SoC Serpent Lake w/RTX in 2028,

- Arc is continued in mobile, professional, iGPU, and DC GPUs,

- Arc team is just fine, Intel cut some jobs but not "the whole team".