[aurareturn]

It isn't that good for local LLM inferencing. It's not designed to be as such.

It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for.

Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.

[jamesblonde]

Given that most of Nvidia's enterprise software products are all single server designed to run on DGX boxes, like NIMs, this makes sense.

I am still amazed at how many companies buy a ton of DGX boxes and then are surprised that Nvidia does not have any Kubernetes native platform for training and inferencing across all the DGX machines. The Run.ai acquisition did not change anything, as you leave all the work to the user to integrate with distributed training frameworks like Ray or scalable inference platforms, like KServe/vLLM.

[teleforce]

If I understand correctly the DGX is for the development, and the AGX Thor is more geared toward local LLM inferencing [1],[2].

[1] (Updated) NVIDIA Jetson AGX Thor Developer Kit to Launch in Mid-August with 2070 TFLOPS AI Performance, Priced at $3499:

https://linuxgizmos.com/updated-nvidia-jetson-agx-thor-devel...

[2] AAEON Announces BOXER-8741AI with NVIDIA Jetson Thor T5000 Module:

https://linuxgizmos.com/aaeon-announces-boxer-8741ai-with-nv...

[spaceywilly]

What is the value proposition for buying one of these vs renting time on similar hardware from a cloud provider?

[xmichael909]

I don't think there is one. Honestly this version 1 is dead on arrival.

[arresin]

I wish I could run Linux on them (the m5)

[hoppp]

There was an arch linux version that supports apple silicon

[porphyra]

Asahi Linux has long switched from Arch to Fedora (though a janky Arch version still exists). But they don't support anything newer than M2.

[kirillzubovsky]

Fascinating that we didn't have to wait too long. Apple announced M5 this morning. Does it compare though?

[NaomiLehman]

because of possible hardware-accelerated matmul in GPU cores?

[christkv]

Massive memory bandwidth for the most part. M3 ultra had like 810 Gb/s vs ~300 for the DGX Spark. Also you can get up to 512 GB memory with a 256 GB config as well

[aurareturn]

Yes. Matmul in M5 GPU, memory bandwidth, consumer/prosumer friendly OS, and they are just excellent portable laptops.