[ein0p]

Amazon's chips at this point are marketing for Amazon. I've seen the benchmarks, they're not quite ready for serious use yet. I suspect Anthropic got a good discount on GPUs in return for using Amazon's own chips in any possible capacity (or maybe just for the press release claiming such use). The only real alternative to NVIDIA on the inference side that you can actually buy hardware for is Intel Gaudi which costs less and performs rather well, but everyone seems to have written it off, along with Intel itself, and it's not available in any cloud last I checked. On the training side there's really no alternative at all - PyTorch is the de-facto standard, and while there is PyTorch XLA, it's even less popular than Jax, which is already like 20x less popular than PyTorch. Bottom line: capable Jax engineers able to optimize distributed Jax programs on TPUs are unobtainable unicorns for anyone but the top labs and Google itself. Note that the training side has significantly different requirements than inference side. Inference side is much simpler to optimize and wring the performance out of.

[mike_hearn]

Yes I've been expecting AMD to eventually get inference working because it's so much simpler. Supposedly Meta do use some AMD for inference. It's sad that you can implement llama inference on the CPU in a few thousand lines of Java yet somehow AMD isn't cleaning up there.