Hacker News new | ask | show | jobs
by stri8ted 90 days ago
48 GB is not consumer hardware. But fundamentally, there are economies of scale due to batching, power distribution, better utilization etc.., that means data center tokens will be cheaper. Also, as the cost of training (frontier) models increases, it's not clear the Chinese companies will continue open sourcing them. Notice for example, that Qwen-Max is not open source.
3 comments
zozbot234 90 days ago
Nothing obviously prevents using this approach, e.g. for 3B-active or 10B-active models, which do run on consumer hardware. I'd love to see how the 3B performs with this on the MacBook Neo, for example. More relevantly, data-center scale tokens are only cheaper for the specific type of tokens data centers sell. If you're willing to wait long enough for your inferences (and your overall volume is low enough that you can afford this) you can use approaches like OP's (offloading read-only data to storage) to handle inference on low-performing, slow "edge" devices.
link
WesolyKubeczek 90 days ago
It is consumer hardware in the sense that Macbook Pros come with this RAM size as base and that you can buy them as a consumer, without having to sign a special B2B contract, show that your company is big and reputable enough, and order a minimum of 10 or 100.
link
m-hodges 90 days ago
> 48 GB is not consumer hardware.

It’s a MacBook.

link
kelnos 89 days ago
Technically that's correct (which as we all know is the best kind of correct), but really, how many consumers are buying a high-end MacBook Pro with 48GB or more of RAM? That's a very small percentage of the population. In these kinds of discussions, "consumer" is being used as a proxy for "something your average home laptop buyer might have". And a 48GB MBP is not that.

I know it's annoying, because a 48GB MBP is indeed technically "consumer hardware", but please understand the context and don't be pedantic. You know what the GP meant. (And if not, that's... kinda on you.)

link
m-hodges 89 days ago
> but please understand the context and don't be pedantic.

The context is this is something I can pick up at an Apple Store and not some rig I have to build with NVIDIA cards.

I led with:

> get closer and closer to consumer hardware

I think this demonstrates getting closer, whether you think a MacBook is consumer hardware or not. But I'm the one being pedantic.

link