[FuriouslyAdrift]

I work for a tiny little company ($150MM annual rev with 9% net) and we are already looking at dropping $100k on hardware to run local models because, for us, they're "good enough."

Our estimated spend for AIaaS would exceed that cost in less than a year.

In a few years, there will be hardware capable of running frontier models good enough for most things at accessible prices for even tiny companies.

[simplyluke]

Yeah, that's the part that just seems to be wildly under-discussed to me.

If open source models are ~3-6 months behind SOTA, and ~opus4.6 capabilities are good-enough for product market fit, do the frontier labs have half a decade to catch up on their prior burn?

AI cost ballooning faster than companies can afford is becoming a very common topic in my circles right now. The era of "I'll pay infinitely more for marginal gains" is over from what I can tell.

[an0malous]

> If open source models are ~3-6 months behind SOTA, and ~opus4.6 capabilities are good-enough for product market fit, do the frontier labs have half a decade to catch up on their prior burn?

They know they do not and that’s why they’re all trying to IPO right now, so they can pass the bag to consumer investors

[WinstonSmith84]

More correlation, if more correlation was needed:

1- SpaceX + Tesla + xAI merger / IPO while Musk was vocal against IPO for about a decade

2- Warren Buffett cash at record highs

Someone got to be exit liquidity

[londons_explore]

The printing press was good enough for product market fit back in the 1700's. But now it isn't.

Last year's AI models will be the same. Do you want to spend 3 hours prompting free AI to fix your code or 1 hour prompting AI you paid $20 for?

[an0malous]

That's only if these AI companies can keep improving their model performance faster than open source options can keep up. I don't think performance will keep scaling with more training data, and even if it does they've likely already used the entire history of content created by humans for training. Everything points towards diminishing returns in an increasingly crowded space of competitors, there's no other reason for these companies to be rushing to an IPO if they felt secure in their market positioning.

[doug_durham]

Open source models that you can run locally are much more than 3 to 6 months behind. 6 months was the November inflection for Claude. No open source model is as good as Claude Opus 4.6.

[jobs_throwaway]

It depends what you mean by locally. I don't foresee running a model on my laptop anytime soon to power a coding agent. Far more likely is an infra team at my company operating an open source model on cloud infrastructure. When they're already paying $1000 / month / dev, it starts to pencil pretty quickly.

[wrsh07]

Is there any open model as good as opus 4.6 at any price?

[overfeed]

How many problems require Opus-4.6-level performance? The "I'll accept nothing but the very best model for any task" thinking is perplexing to me.

People got a lot done before Opus 4.6. In 6 months, would you be dissatisfied by Opus-4.6-level open-weight models, just because Opus 4.8 will be out?

[strix_varius]

Not OP but I've been thinking about this a lot (like everyone ha) and I think my answer is, yes?

I hope there's a "good enough" point but I don't think we're there yet. Like for me hardware got good enough several years ago. But while opus 4.7 is really good compared to everything else, it's not so good that I would use it at a discount over whatever is available in a few months. The improvement in quality, speed, and daily frustration is worth it to me... Spoken as someone whose employer is footing the bill, so take that with a grain of salt.

I want to run my own local models, but I don't think that's feasible without lots of frustration until a few generations of frontier models are so good that they're almost indistinguishable for common tasks. Kind of like how MacBook pros have been for a while.

[wrsh07]

I'm very happy to have multiple sessions open (and do) and switch between fast and slow models, and if there were a batch mode in codex or Claude code I would use it. (Just like I sometimes use codex fast mode)

But at the moment, I can't imagine why I wouldn't be spending the majority of my time with the best models. I'm spending a lot of time with them! Reducing the number of back-and-forths is extremely valuable to me.

I expect in two months I will still want to spend >80% of my time prompting the best models, and that's true if I were spending my own money on hobby projects, too.

[JohnBooty]

    would you be dissatisfied by Opus-4.6-level open-weight 
    models, just because Opus 4.8 will be out?

Well, I see what you mean, but two big concepts...

1A. Models get stale pretty quickly w.r.t. new developments that occur past their cutoff date. "But you can just keep them current by linking them to never documentation, etc!" Well, no, you sorta can't -- at least not in perpetuity. Those search results fill up your context window real quick. So that gets unsustainable real quick.

1B. Even when your context has plenty of free space, the results you get from "here's a link to the documentation for this new framework that released after your cutoff date" absolutely pales to the results you get from knowledge that is fully baked into the trained model as opposed to your context window. For one thing, that documentation link you pasted into your context might link to... a dozen code examples. Whereas if that was baked into the model itself, the model might have been trained on many thousands of examples in Github etc.

2. It's also a reality that most professional engineers have to keep up with their peers and competitors. We can maybe say it shouldn't be that way, but it is. So if $SOME_NEW_MODEL is significantly better than 4.6... and my peers and or competitors are using it, then yeah I might but really feeling the need to match them. And I'm not even necessarily talking about some kind of cutthroat dog-eat-dog stack-ranked workplace.

These limitations aren't relevant for all use cases or careers but they're hiiiiiiiighly relevant for professional software engineering.

[Paradigma11]

As long as the improvement is vastly more valuable in my time than the added cost I will always use the best model. I think it depends on your situation and tasks what makes sense.

[FuriouslyAdrift]

Kimi 2.6 probably. Needs over 300GB of GPU memory to run (1TB for for full capabilities) so either a 4x A100 or 8x A6000 would do it.

A $50k - 100k rig could do it and an entire company would be able to use it a full speed.

[chillfox]

No, but the big open models are on the level of Sonnet 4.6, which is very good for most problems.

The people who are claiming Opus level capability does not have sufficiently complex problems to see the difference.

[irthomasthomas]

And neither side brings any evidence ...

[raxxorraxor]

For coding don't think so, but they are very close. I code with sonnet mostly because I think opus is just useful if you fail to dissect problems adequately, but anyway.

Kimi is close for example regarding SWE bench for code. For reasoning there are open models that surpass opus by quite a margin already.

[simplyluke]

> that you can run locally

That's doing a lot of work here.

The future I see isn't most companies buying hundreds of thousands in hardware to run models, it's them adding a line item to their AWS bill. Inference costs on the larger hosted open source models are dramatically lower than the frontier labs API pricing.

[teiferer]

The future I'm seeing is AI coprocessors running inference locally in most devices that today have a CPU. Just look at how powerful your mobile phone has become compared to your desktop computer 15 years ago and compared to a main frame 30 years ago.

The days of requiring a data center to run anything resembling opus 4.6 are already counted. (But the industry will fight hard to get people to keep paying the Claude tax.)

[simplyluke]

I'm already running a google TPU over USB on an otherwise very cheap board to do local computer vision on a front-door camera since I wanted to get away from Ring and other cloud services for that use case.

And yeah, that may be the ~decade world, but we're in the mainframe era of the frontier models. It's going to be more economical for basically any consumer, and most businesses, to pay someone else to host models for quite a while.

[lelandbatey]

A gaming PC can already host models that perfectly serve casual users who just want recipes, todo tracking, picture identification, etc. E.g. Qwen 3.6 35b which will run on a $650 GPU at 75 t/s (Nvidia 1660 ti 16GB).

Said model will also run as a tool-calling coding model excellently (it's no Opus, but for a thing that once set up is just the cost of energy, it's incredible). It can type faster than you can, probably 10x faster, so with guidance it'll make you faster. And it's free.

It's here. If folks want ChatGPT without a subscription, they can have it today on their computer. The only money to be made is in the high end models doing "serious business" work spanning 1M+ token contexts and massive uncertainty. Everything else is already set to be eaten by today's local models.

[dom96]

Curious why you went for a custom solution. I am aware of at least one company that seems to ship devices with local computer vision (Reolink).

[gedy]

> But the industry will fight hard to get people to keep paying the Claude tax.

I bet this will ironically be couched in "safety" reasons or regulation to get anti-AI folks on board, even if it favors the large incumbents.

[selimthegrim]

Counted but not yet numbered?

[enaaem]

Even when run on datacenters, it would be like current day webhosting. It is hyper competitive and it will be a race to the bottom. There is money to be made but not as much as investors hope. There will be datacenters in random countries like Kazakhstan because some oligarchs have found a free energy glitch (like with bitcoin mining).

[ai_fry_ur_brain]

Magical thinking. I guess if your phone is going to have 128gb of dddr5 then sure. You people fundamentally don't understand the memory requirements for running inference. Your cute local models seem good enough because you have no standards and anything an LLM produces seems like magic to you.

[teiferer]

> Magical thinking. I guess if your phone is going to have 128gb of dddr5 then sure.

Why would it not? The typical new phone today has 16gb of RAM. 20 years ago that was somewhere around 32mb. Factor 512. It's not hard to see that we'll get there rather soon, especially if there is an application that provides demand.

> You people fundamentally don't understand the memory requirements for running inference.

You seem to be overlooking how fast things change in this industry, especially if tons o money can be made as a consequence.

> Your cute local models seem good enough because you have no standards and anything an LLM produces seems like magic to you.

Please don't generalize. I'm an expressed AI skeptic and have to deal with the bad consequences of AI slop every day. But you can't deny that there are enough applicationn areas where people have use cases and those will be much easier if things don't need a few round trips to a data center that sucks all the electricity and water out of neighboring communities.

[apocalyptic0n3]

> it's them adding a line item to their AWS bill

That's the future Amazon sees too. We just had a week long session with the AWS team and they pushed that to us multiple times.

[majormajor]

Buying "hundreds of thousands in hardware" sounds like a lot but many companies - especially software companies - already do that if they have 100+ employees.

Running software in the cloud gives you certain reliability and scaling advantages that would be very hard to replicate locally. Running some code agents in the cloud vs local hardware, if the local hardware gets "good enough," breaks the other way - offline usage, alone, would be hugely valuable to many people and companies.

It'd be very interesting to see where various players would decide to make a call "local is good enough" though. Buying the hardware isn't a small bet, if it's not something that ends up as part of your standard computer.

[PeterStuer]

Many business tasks do not need the latest frontier models. I have a production system running since early GPT-4o. It now runs with GPT-5.2, not for improvements, but because it is cheaper. I could invest in switching to a local model, I tried and it works well enough, but api costs for this task are so low, it barely scratches $30/month. So I am using the local machine for other things and leave the inference on OpenAI, for now.

[PunchyHamster]

But one will be in few months. And then you have choice of paying say $100k for hardware and pay just power cost (or pay someone to do that for you), or pay way, way more for your team to have access to marginal improvement.

And 5% worse model for 10% of the price of the bleeding edge will be worth it for majority of people

[lukeasrodgers]

This project argues that with appropriate harness, the performance gap between frontier and much smaller open weight models shrinks dramatically: https://github.com/antoinezambelli/forge. I haven't kicked the tires yet.

[myaccountonhn]

I've been doing my work with OpenCode Go, with Kimi2.6. It is not as good as Claude Opus, but it's good enough to get the job done, and I never run out of tokens.

[overgard]

I keep hearing about this "inflection", but it feels extremely exaggerated to me. And yes, I was using it at the time. It got incrementally better, it wasn't that amazing.

[simplyluke]

I think the bigger shift was harnesses and the two ended up somewhat commingled in people's minds.

Claude code was a lot of people's introduction to using coding agents that could do a lot more than copy-pasting from a chatbot or autocomplete.

[noman-land]

The tool usage + skills got markedly better and so did the thinking cohesion. Add 1m context windows and it was a very noticeable shift.

Opus 4.6 quality for local inference would be revolutionary.

[viking123]

1m context is garbage

[nazgul17]

It's just a metric. If it can find a needle in a 1Mtok haystack, then it's likely good at coding within a 200Ktok context (or whatever, insert your number here, I'm just trying to make a point)

[applfanboysbgon]

Opus 4.6 is a February model. Every time this subject comes up it seems like people post intentionally misleading things and move the goalposts.

The goalpost we've been bludgeoned with over and over again is that, in particular, Everything Changed in November 2025. That GPT 5.2 and Claude 4.5 were the inflection point. That is actually 6 months ago. And DeepSeek 4 is already there.

> run locally

You can't run DeepSeek locally on consumer hardware[1], but you can on enterprise hardware, and enterprise spend is the subject of this conversation -- and even if you aren't self-hosting, it doesn't matter, because you can just get your inference from one of the the many companies serving DeepSeek, who trivially undercut the pricing of OpenAI/Anthropic because they didn't have to spend hundreds of billions on training frontier from scratch but instead only invest in supporting inference, which is already profitable.

[1] Since this misconception comes up all the time, I'll go ahead and pre-empt it: no, training a 32b parameter model on outputs from DeepSeek and running that locally is not "running DeepSeek", despite the hundreds of stupid articles and Youtube videos making that idiotic claim that they're running it on a 5090.

[simonw]

> You can't run DeepSeek locally on consumer hardware

Maybe not DeepSeek v4 Pro, but I've run DeepSeek v4 Flash on my 128GB MacBook Pro using antirez's carefully quantized https://github.com/antirez/ds4 and it's impressive.

[applfanboysbgon]

Oh sure, yeah, that's nothing to sneeze at either. I think unqualified "DeepSeek" should generally refer to the main model, though, especially in the context of GPT5.2-grade quality.

[zozbot234]

> You can't run DeepSeek locally on consumer hardware

I'd qualify that by writing that you can't run it with ordinary, real-time speed and throughput. If all you care about is slow and high-latency inference, there's no reason why that shouldn't be feasible even on the cheapest miniPC around, as long as it can literally store the model weights and keep around the (rather small) context.

[damnitbuilds]

To be relevant to this discussion, models running on reasonably-priced local hardware do not have to be as good as the best.

They just have to be useful enough that companies don't need the best.

They are.

[londons_explore]

Deepseek v4 pro is damn close to Claude 4.6, and whilst you'll pay quite a lot for a rig able to run it, it is open source.

[_3u10]

Kimi is better.

[svara]

There's still a lot of room for the best models to get better at coding .

Your argument rests on the "for marginal gains" part but it's really not clear that the gains are marginal in the foreseeable future.

[simplyluke]

This is totally valid and I don't agree with the downvotes you're getting. Someone coming out with a 10x improvement is possible and would change the game immediately. The thing is, we really have been seeing marginal gains with shifting leaders in who's got the "best" since GPT3, and at least as a user of these tools that pace has been slowing, not accelerating. Subjectively it feels like we're in the back half of an S-curve.

We're 3.5 years into this current AI wave, and a lot of the valuations have been predicated on what you're arguing here -- that essentially should one of the labs make an order-of-magnitude improvement or hit escape velocity on recursive self-improvement they'd become the most powerful economic chokepoint in history.

The reality has been that given access to compute + capital all of the labs can stay pretty competitive with each other. Someone does a bit better on coding, someone else does a bit better on tool calling, and then they swap after each spending another $100bn.

The market looks like a commodity market where the commodity is intelligence, not a winner-take-all market with massive margins. Plenty of people get rich in oil and airlines, but they notably don't tend to be the innovators long term, they tend to be the operators. Obviously if the machines become sentient tomorrow, turn on their masters, and hit world-dominating intelligence, that assessment changes, but after several years of that narrative while objective reality looks quite different I think the more sober voices are starting to gain a foothold.

[svara]

I agree with most of what you're saying, but I think the point I was trying to make wasn't as high-flying as you and others understood it.

I'd pay a premium for even just a model that's 20% better, no ASI required, and I think a lot of people would. I wouldn't call that marginal, if it means I'm getting frustrated on 20% fewer tasks.

A recurring pattern that I've seen in myself and others is to at first be very impressed by a new model's coding capabilities, and then desensitize quickly and start being frustrated by the shortcomings.

[simplyluke]

> I'd pay a premium for even just a model that's 20% better

The point I'm making is that I think we're rapidly hitting levels where corporate buyers aren't willing to pay multiple-times-more for marginal gains, and I expect that to become more the case over time, not less. You, and a small % of other power users in the market might tolerate a $400/month pro-supreme-plan for access to Mythos or whatever, but I don't think that's going to scale up in quite the same ways we've seen so far.

Even a year ago paying multiples times more for a 50% gain was very sensible for a lot of workflows. But if we're getting to "good enough" for things like coding, justifying to your CTO/CFO why the org should go from spending $1m/year to $5m/year for a 10% higher hit-rate on one-shot prompts from the engineers is a much tougher sell.

[yfw]

What? The gains between gpt4->5 seems to be marginal. No phd level discoveries here

[simonw]

The leap from GPT-4 to GPT-5.5 has been astounding in my opinion. There is no way GPT-4 could run a coding agent harness like Codex at even a fraction of the quality that GPT-5.5 does.

[anon373839]

I don’t think that’s exactly indicative of GPT-5.5 being an astoundingly more intelligent model, however. An alternate interpretation is that GPT-5.5 was trained on tool usage/harness patterns and has been optimized for this use case.

I remember that even when GPT-4 was king, the Gorilla paper showed that Llama 7B could be fine-tuned to outperform GPT-4 on tool calling.

On domains that don’t involve agentic tool calling*, I haven’t found the frontier to have advanced that much.

Edit: I should broaden this to domains that naturally lend themselves to RLVR training. Models are drastically better at math now.

[baq]

None of this matters in the product: it either is capable of agentic loop workflows or it isn’t. A 10% improvement in probability of single task success makes or breaks the use case.

[imtringued]

For me any of the codex models run circles around the non codex models for codex usage.

I'm not sure why you're so obsessed with the non-codex versions

[swalsh]

Open source models, especially qwen are pretty dang good. But its not opus 4.6, the evals dont tell the full story. I question the assumption open source models are 3-6 months out.

[Ucalegon]

Its not just about the quality of output, but you also can finetune them to proprietary needs, if the skillsets are their internally, to make them better without governance risks. So being SOTA doesn't matter as much, since generalized tasks are not what matter most to companies, its the specialization relative to business need or internal datasets.

[oblio]

To make an extreme comparison, desktop Linux was originally supposed to happen in 1999.

[simplyluke]

Maybe I misspoke by saying open source.

The larger point I'm making is I think models are rapidly becoming commoditized. There is probably a small market long term that's willing to pay 10x for 10% marginal gains, but the majority of the buyers in the market will be economic and we're likely to have a lot of folks willing to spend 1/10 the cost for 90% of the performance, and plenty of companies that haven't raised hundreds of billions-trillions who can provide that.

A lot of the frontier labs valuations has been based on an assumption that 1-2 companies would get break-away intelligence that basically made them economic chokepoints indefinitely into the future. The reality that's becoming increasingly clear is that model quality is a pretty linear function of (cash burned - ability to copy other's homework) and the economics are starting to look a lot more like airlines than online advertising.

[grttq]

Lets go one step further.

The economics of airlines are such that they generally earn a return on capital less than cost of capital.

I think this is exactly where we are heading and OAI-Anthropic are the concordes.

[extraextra]

Not OP, but it is a known fact that the cumulative profits of the airlines industry (in US) over it's history has been basically 0. We can say that essentially airlines are in business to support other businesses. I believe this is what OP might've been referring to.

[w29UiIm2Xz]

If only the AI era was born in ZIRP.

[sailfast]

Better now than ZIRP for me - at least people are asking timid questions about the unit economics and how long the runway is _early_ while also spending absolutely insane amounts of money on this bet. During ZIRP, these companies would have turned down any investor asking questions. Less contagion when rates aren't zero hopefully? :grimace:

[mschuster91]

The size of the AI bubble and the IOUs being passed around like a hot potato already dwarfs the real estate bubble preceding the 2007 crash.

If we still were in the ZIRP era, busting the bubble would certainly kill off the world's economy for good simply due to its size.

[vessenes]

You have to think about why open models are behind. Exfiltration is a big part of it. So you could change the Nash equilibrium by increasing your security, or other multilateral approaches.

[drumdance]

For give my naiveté, but who pays for the training of these models?

[EvanAnderson]

> ...we are already looking at dropping $100k on hardware to run local models...

Just think how much further that $100K would have gone if the hardware market wasn't so screwed-up.

Anecdote: I priced-out adding 1TB of RAM to a four node cluster a couple months ago. The cluster was purchased in fall of 2024 w/ 4 nodes, each with 256GB RAM. The nodes cost just over $14K apiece back in 2024 (entire box, not just the RAM).

Dell wanted >$90K a couple months ago to add 256GB to each node.

[cyberax]

> Dell wanted >$90K a couple months ago to add 256GB to each node.

RAM is expensive, but not THAT expensive. I just bought 128Gb for about $5k for our build cluster (it's not even for AI, sigh). Even if you need larger-sized DIMM sticks, it's still going to be in the vicinity of ~15k tops.

[EvanAnderson]

It was crazy. I found the part on the open market for a lot less but the edict from the Customer was to buy from Dell to keep the support entitlement intact. That inflated the price to an astronomical level to be sure.

I haven't had problems w/ Dell support and 3rd party memory, personally, but given the machines' application I understood the concern.

[rstuart4133]

I get the impression the hive mind hasn't come to terms with the point that a model is optimised for certain tasks. It's like having someone ask you "is that a good hammer?". Good for what? There are claw hammers, sledgehammers, ball-peen hammers, club hammers, mallets, .... Yes, in a pinch, they can all bang in nails, but you wouldn't choose a dead blow hammer for that if you had a choice.

The Gemini Flash is very good at searches. Just about any low end model can toss out a poem. All the higher end models (open source and otherwise) seem to be able to churn out code that passes tests. The smaller, "less capable" ones are much faster at it, which means in the hands of a skilled practitioner are the best choice for that task. But they rapidly fall apart where there isn't a hard source of truth (like a good test suite) to grind against. Because of that you have to use a bigger model for bug finding. In that task the open source models tend to fail on larger code bases, where something like Opus still shines. I gather Mythos is an absolute monster, and unparalleled, and unavailable. I'm sure one of the reasons for that is it's so expensive to run.

Or to put it another way - you don't use a 100 tonne crane to pick up the shopping. And ... the smaller models will happily run on in-house hardware. You may not do it today because of the current DRAM price and integrated NPUs have just started shipping, but in 5 years time models will be running on your phone.

[Npovview]

Yes exactly, we will have specialized models soon. These will be trained with plugin architecture with a core reasoning model asking plugin models to do stuff on its behalf. I don't need chinese or russian knowledge in my workflow.

[safety1st]

Yes 100% this. A lot of people keep talking about how OpenAI and Anthropic will need to raise their prices. What is less discussed is how they CAN'T raise their prices because competition exists, and sure it's not SOTA, but it's literally an order of magnitude cheaper in many cases and the drive to figure out how to make it work well enough is going on right now (and will only intensify when the SOTA models raise their price).

It's a given that the SOTA models need to raise their prices. It's also a given that they can't. The more they raise the more customers will move to their competition.

So what happens next? Well I think it will suck horribly if you can't move off of SOTA sooner or later, because the Big Two are going to lose customers, and therefore have to raise prices on the locked in customers even more than these projections suggest.

Beyond that if you're looking to start a business, figure out how to use cheap models in new scenarios. Build software which does that and license it. This is kind of contrary to the idea that you shouldn't over optimize for deficiencies in the models that will likely go away in the next generation - for instance a lot of problems were solved when context windows got way bigger. So it's a thin line to walk but I think it's there because a lot of orgs are using Claude today for pretty basic tasks.

The dev who's addicted to SOTA models honestly is going to have to settle for less or get totally screwed. Most applications within business from what I see aside from complex research do not require SOTA. They summarize, they classify, they transform, and doing that accurately has been cheap for a while.

[extraextra]

I'd qualify your point that Anthropic and OpenAI can't raise prices, that is as of right now. Once the industry will go through a phase of consolidation and the bigger players will have some moat around their product, they'll have more pricing power.

Your last point is common sense in my opinion, I agree with it. At the end of the day most employees are (by definition) of average intelligence and most businesses are average in complexity. Thus, it is logical that average tools (AI models) should do the job for most people and most businesses.

[MASNeo]

On prem AI makes sense for more than just the cost. More control, IP, model improvements you can keep, data privacy to name a few. People will realize that AI is not like compute the moment they get their own knowledge sold back at a premium.

[fragmede]

What are the advantages to on-prem for a company that's already in the cloud and trusts it with their IP? That company can just rent GPU instances from the cloud if they want to train/fine-tune their own models and keep avoiding CapEx.

[amelius]

> People will realize that AI is not like compute the moment they get their own knowledge sold back at a premium.

But what if your competitors sell their knowledge to AI companies?

Then you're still screwed.

[stopachka]

I don't quite understand, what would 100K buy you?

AFAIK you would get about ~5 concurrent users, with a max context window of ~128K tokens on the larger models.

This wouldn't be good enough for coding -- are you guys thinking of using it for something else?

[bendews]

By my calculations 100k could get you 18 5090's + compute to host them, or 18 96gb Mac mini's. You can get a lot of context window and users out of that setup.

[FuriouslyAdrift]

Gigabyte 4x AMD Instinct MI300A rack server (512GB GPU RAM total)

Roughly equivalent to 4x H200's for less than half the price.

Vaguely around 60k tokens per second...

[cmdrk]

Do you think this will be a trend for larger companies as well?

The decadal move to all-cloud-all-the-time killed off in-house hardware teams while the C-suite chased their OpEx dreams.

It would be interesting if we come full circle on this.

[fragmede]

I doubt it. Companies that have moved to the cloud are already trusting the cloud with their IP. You can rent time on a high end Nvidia system from various clouds. OpEx means there's no write down in three/five years as that system goes out of date so it would only make sense if the performance/$ is there, or the company is highly protective of their IP and doesn't trust the cloud, at which point they're not on the cloud anyway.

[fittingopposite]

Agree. You have these tipping points when a model is good enough to do some task. Yes, a better model will further improve your capabilities but the unlock is at a certain intelligence level. We see this also with humans. People with very low intelligence can't learn to read. Once you cross a certain threshold of intelligence you can learn to read. More intelligence doesn't really help you in the task of reading. A person with an IQ of 160 is not substantially better in reading than someone with an IQ of 85. If your IQ is 50, you might not be able to learn to read at all.

[teaearlgraycold]

Have you considered that a smarter person will understand what they have read better?

[physicsguy]

Depends on the task and the writing though doesn't it?

There's not that much depth in a lot of 'everyday' writing. For many tasks that means that you don't need to be hyperintelligent - reading a recipe or a shopping list, reading a newspaper article, etc.

[mv4]

I configured a dual DGX Spark cluster, and it's certainly "good enough" for my agentic and coding needs.

[datadrivenangel]

what models are you using on that? My experiences with apple hardware have convinced me that it is not really good enough for coding locally.

[girvo]

DeepSeek v4 Flash, various quantised versions of Kimi K2.6, MiniMax 2.7, Qwen 3.5 “full sized, with a dual spark setup you can fit some decent setups on here

My single spark has me running Qwen 3.6 27B and antirez’s specially quantised DeepSeek v4 Flash (which is shockingly impressive)

[Tepix]

Kimi K2.6 does not run well on 256GB.

[zozbot234]

Have you tried it? It would be slow for sure, but the main limitation AIUI would actually be storing the context in RAM - models like Kimi and GLM have high demands there which limit your ability to get meaningful aggregate throughput via large batches.

[Tepix]

No need to try really. 1100b weights with 256GB RAM that‘s less than 1.8 bits per weight if you want a little bit of context.

How is that supposed to give good results?

[girvo]

True, I might be thinking of some of the communities four-Spark clusters for it; it’s already int4 right?

[Tepix]

Yeah, the default quants are 595GB. Even four Sparks would require a quant lower than 4bit

[irishcoffee]

It isn’t the models, it’s the closed api and the tooling associated with it. It’s driving me crazy how not-talked-about this is.

[klausa]

You can point both Codex and Claude Code at a local model and they'll work just fine. Codex even explicitly supports that as a feature! [1]

With a nice UI on top, for the desktop app too: [2]

[1]: https://developers.openai.com/codex/config-advanced#custom-m...

[2]: https://docs.ollama.com/integrations/codex-app

[datadrivenangel]

As in the coding harnesses?

[irishcoffee]

If I could leverage the same closed api VSCode uses, the entire moat is drained.

What you call harnesses I call… bullshit?

[mv4]

Minimax M2.7.

[alex_suzuki]

I’m curious: are you spending on beefy developer machines, or some kind of shared local inference server? Would be interested to know more if it’s the latter.

[irishcoffee]

I am aware of at least a handful of companies doing the latter. I don’t work for them and cannot speak to their setup.

[awesome_dude]

> In a few years, there will be hardware capable of running frontier models good enough for most things at accessible prices for even tiny companies.

I was going to say - the models are just going to keep growing at a pace exceeding the pace of hardware pricing/availability

But then I realised that, far more likely, there will be a plateau reached (again) where nobody is seeing gain, and at that point hardware will catch up

[slashdave]

It might be possible that in a few years someone will be able to engineer a reasonably priced machine to run today's frontier models (hint, your price is an order of magnitude off). However, they won't be able to run the frontier models that will exist in a few years.

[AndrewKemendo]

That’s exactly where the market is heading and it’s going to have to reckon with this fact

My guess is there’s gonna be some legislation or something “you can’t share anything over this level of complexity” and I think that that’s what a lot of that mythos rattling was all about

[arbuge]

> In a few years, there will be hardware capable of running frontier models good enough for most things at accessible prices for even tiny companies.

What makes you so confident about this prediction? Hardware costs haven't exactly been cratering recently.

[sofixa]

.> Hardware costs haven't exactly been cratering recently.

No, but local models have been booming in performance/quality improvements. The RAM shortage won't last forever (more supply will come online when if demand doesn't diminish), and then the math would be pretty easy.

[soonnow]

I see even smaller companies ask for that. Less about cost, European companies have a deep mistrust of US companies at the moment. I see companies a tenth of your companies size ask about local models.

[wyager]

> there will be hardware capable of running frontier models

The current frontier? Sure. The frontier then? No - obviously that frontier is going to keep consuming available datacenter compute capacity, which will be better

[physicsguy]

My much larger company has got people already using various models through Bedrock because the Claude and OpenAI limits are too harsh and it's too expensive.

[try-working]

What about using DeepSeek API? Practically free.

[disiplus]

same, but you need more then 100k of hw to run something like kimi k2.6 for a bigger team. on the other hand there is a ds4 flash that you can run on a macbook with 128gb ram. an that one is perfectly usable for a lot of tasks.

https://github.com/antirez/ds4

[FuriouslyAdrift]

I think the quote came out to $107k. 4 AMD MI300A's. Around 60k tokens per second, 512GB of GPU memory.

https://www.gigabyte.com/Enterprise/GPU-Server/G383-R80-AAP1

[disiplus]

Which model are you running ?

[nonethewiser]

What models? Last I tried different local modals there was a pretty big difference from frontier.

[ai_fry_ur_brain]

You people are delusional. How many times a day am I going to read this fiction of "good enough in a few years for most things".

There are physical limits to how much you can compress data and how much is needed for a capable model. If by hardware capable for running SOTA you mean a 7 figure investment for a company, than sure. But how come these companies didnt do the same thing for cloud? There's been this option for self hosting infrastructure for a decade but companies don't use it, they pay AWS.

[amelius]

Eh, one question. Where do you intend to buy the hardware if datacenters take over the market?