[spwa4]

> but even using 16GB sticks, 24x16GB is already 384GB.

Question: you need 16GB sticks because they're the smallest doublesided ones, which you need for maximum BW, right? Otherwise why not 8G?

[AnthonyMouse]

If 8GB sticks of registered DDR5-6400 even exist, they're not common.

[ethbr1]

That was because memory used to be cheap. Will be curious if smaller capacities come back en vogue as a cost cutting mechanism.

[AnthonyMouse]

Using a server system designed for having hundreds of cores and TBs of RAM for LLMs because it has a medium-high amount of memory bandwidth is a hack for enthusiasts who want to take the price/performance trade off to run big models without paying for enterprise GPUs. The market for 8GB RDIMMs would be specifically that, since the more typical server workloads that need that amount of memory bandwidth also want the larger memory sticks (e.g. database servers, systems running hundreds of VMs), or aren't bounded by memory bandwidth to begin with and then don't need to populate all the channels.

And if you wanted that market segment then what you'd really do is produce a consumer GPU with 128GB of RAM.