[dagmx]

At what point does the on device stuff eat into their market share though? As on device gets better, who will pay for cloud compute? Other than enterprise use.

I’m not saying on device will ever truly compete at quality, but I believe it’ll be good enough that most people don’t care to pay for cloud services.

[whimsicalism]

You're still focused about inference :)

inference basically does not matter, it is a commodity

[dagmx]

You’re still focused about training :)

training doesn’t matter if inference costs are high and people don’t pay for them

[whimsicalism]

but inference costs arent high already and there are tons of hardware companies that can do relatively cheap LLM inference

[dagmx]

Inference costs per invocation aren’t high. Scale it out to billions of users and it’s a different story.

Training is amortized over each inference, so the cost of inference also needs to include the cost of training to break even unless made up elsewhere

[rowanG077]

That makes no sense. Inference cost dwarf training cost if you have a succesfull product pretty quickly. Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1.

[whimsicalism]

> Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1.

Stack enough GPUs and any of them can run o1. Building a chip to infer LLMs is much easier than building a training chip.

Just because one cost dwarfs another does not mean that this is where the most marginal value from developing a better chip will be, especially if other people are just doing it for you. Google gets a good model, inference providers will be begging to be able to run it on their platform, or to just sell google their chips - and as I said, inference chips are much easier.

[rowanG077]

Chip level is only a tiny part of the story. Training can happen with a big boy variant of "it works on my machine". Inference require a world wide network of GPUs. Chip level is the last thing you will be worrying about.

[menaerus]

Each GPU costs ~50k. You need at least 8 of them to run mid-sized models. Then you need a server to plug those GPUs into. That's not commodity hardware.

[whimsicalism]

more like ~$16k for 16 3090s. AMD chips can also run these models. The parts are expensive but there is a competitive market in processors that can do LLM inference. Less so in training.

[menaerus]

> more like ~$16k for 16 3090s

I don't know where did you get that price from but 1x RTX 3090 is $1,900. 16x is ~$30,000.

> The parts are expensive

Now that we invested ~$30k in GPUs, we only need to find a motherboard that can accommodate 16x pcie4 x16 GPUs, right? And we also need a CPU that can drive that many pcie4 x16 lanes?

Well, none of them exist, not even in the server parts sector let alone client commodity hardware. In any case, you'd need two CPUs so even with this imaginary motherboard we are already entering the server rack design space. And that costs 100's of thousands of $$$.

> but there is a competitive market in processors that can do LLM inference

Nothing but the smallest and smallish models. If that existed then why would you set yourself out building a 16x RTX 3090 machine?

Sorry, but you're just spitting out non-sense.