Hacker News new | ask | show | jobs
by dtquad 670 days ago
Self-hosting LLMs is expensive at scale. It's cheaper to use VC subsidized model inference like the OpenAI APIs.
3 comments

There are plenty of VC-subsidized inference provider which uses open source LLM for much cheaper than OpenAI (which isn't really VC-subsidized at this point but Microsoft-subsidized).
My anecdata is most teams I've talked to say its below OpenAI at scale, and vLLM is a beast. It's interesting to hear the opposite, there's lots of cheaper providers, but the "VC dollars" argument can go "turtles all the way down", I suppose. Still, reality seems to differ.
At "scale"? At what scale?