[hedora]

Pre bubble prices (~= “we stop building data centers with subsidized credit / circular loans / hidden debt”), a 128GB halo strix ran for $1400, and 200-ish watts. Four of those in a cluster will run a 1T parameter frontier model:

https://www.amd.com/en/developer/resources/technical-article...

At 7 months of claude code subscription per node, the cluster pays for itself in 28 months. On a 5 year (60 month) depreciation schedule, you can buy two of those clusters for basically break even, so you get two concurrent request streams (each of which can batch, etc).

The next generation hardware has already been announced, and should ship roughly two Moore’s law doublings later. It’s likely its steady state price is <= $1400 USD (2024), and it is faster.

So, once the bubble pops (because the financial machinations eventually will come to an abrupt halt), and the labs stop buying hardware for data centers, local inference will be extremely practical and cheaper than a subscription.

My main question is, when that happens, will UNIX Surplus be selling inference servers for pennies on the dollar (like after the dotcom crash), or are the power requirements too exotic for home use?

[mft_]

I’m happy to be proven wrong, but the limited examples I’ve seen of clustered Strix Halos are quite slow running large models (ie models too large to fit into the ram of a single machine) due to the slow networking between each one?

[downrightmike]

That is already being addressed with GPUs and special PCIe daughter cards that link them together faster than the PCIe on the MOBO could. SLI but better.

[hedora]

Yeah; the benchmarks + setup guide I linked show that it takes a lot of careful setup to get them to work at all, and then you only get 100's of tokens per second per cluster.

That's why I pointed out the next generation is coming soon. Also, the AMD docs aren't using quantization (as far as I can tell, I only skimmed), which gives a speedup roughly linear in the compression ratio. Algorithms for that continue to improve, so expect a lossless factor of 2-8x on DRAM and throughput, at least.

[mft_]

If I rephrase your earlier post, and based on the AMD post:

If you take four Framework Desktop 128 GB Strix Halo motherboards at a previous low and currently-unattainable price, and ignore the cost of storage, power, networking, and cases, and then take an older model that's just about competitive with Opus 4.5, and then you quantise that model down to Q2_K_XL and ignore the performance degradation below Opus 4.5 this will cause, and then you compare the cost of this with the highest tier of Claude subscription that includes much newer models which are far more able than your Opus 4.5 benchmark, then you might have some sort of break-even inside a year.

Yes, there's a new AMD hardware generation coming, but it's going to be as heinously expensive as the current one has become, and it's still got relatively low memory bandwidth. Yes, there are already newer models than Kimi 2.5, but with the limitations of this cluster (e.g. still needing heavily quantised models to be feasible) you'll get incremental improvements at best.

I'm keen to be supportive and bullish about open/home models, but I worry that this is such a stretch, and the two options in your comparison are so incomparable, you'll turn far more people off than you convince.