[password54321]

Yeah I don't think any of the labs have some secret sauce for intelligence either. It seems most of the advancements are still coming from hardware, making LLMs more efficient and throwing more compute and data at problems. And even those problems still require a lot of prompt engineering: https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98...

[andy99]

The secret sauce is training data. They’re not just taking advantage of more compute (which obviously is necessary but as mentions basically a commodity). They are paying billions to data labelers and making judgements about the nature of the training data they best need to make the product they want. This seems to get pushed aside as a minor point but it’s the primary differentiator of the big labs.

[password54321]

As a I said, compute and data. But LLMs can be distilled, so even their data is not much of a secret sauce.

[reinitctxoffset]

I'm pretty sure at this point that Anthropic is training mixture models (at least in the heavy pre-train) and deploying them dense with explicit loss on thinking trace coherence.

Having a thinking trace that is legible, coherent, and immediately implies the explicit turn output and/or tool use seems difficult if not impossible to reliably get from mixture models.

I predict MoE is a transitional technology, it's got too many problems and the benefits are...kinda grandfathered into the dogma at this point.

[charcircuit]

>I predict MoE is a transitional technology

While scaling laws hold (more weights = better), and time / financial costs are not trivial the incentives are in place to have MoE. MoE means you can have more weights without increasing the critical path of evaluating it.

I am curious what you believe the problems with it that would cause people to prefer using less weights. I'm not following what you mean by MoE can't have legible thinkings trace or tool use when existing models with MoE can.

[reinitctxoffset]

Weights are not created equal: while interpretability is a young field the prevailing view at the moment is that MLP (hence experts) in a mixture model are substantially where dense encoding of factual information resides, attention is even less easily interpreted but it should be uncontroversial that temporal/sequential modeling occurs here.

So it's more consistent with available empirics to say that an architecture can be characterized along a spectrum from fully dense to mixture (a sub spectrum) to Engram-style lookup, and the amount of model power allocated at this point or that will recover different performance profiles.

By far the most stark example of how much performance in reasoning is left on the table is Qwen3.6-27B, which depending on the task, comparison model, and whose benchmarks you believe outperforms mixture models 15-60x larger in total parameter count.

It's badly under-studied (in public) because of the paucity of modern dense models at the near frontier, but even that one data point pretty much rules out the cocktail party version of the Chinchilla-adjacent scaling thesis (which wasn't about modern MoE to begin with).

The "Mixture of Parrots" work is a good jumping off point if you want to get a modern literature review.

[charcircuit]

>the prevailing view at the moment is that MLP (hence experts) in a mixture model are substantially where dense encoding of factual information resides

Yes, because that's where all the parameters are. For reference in GLM 5.2 98% of the weights are for the experts.

>The "Mixture of Parrots" work is a good jumping off point

The paper shows increased performance on knowledge dependent task while having similar reasoning capabilities. This backs up what I was saying about how the weights unlock extra performance without increasing inference costs as much as a dense model would.

>model power allocated at this point or that will recover different performance profiles

While increasing the number of weights makes the model better, where those weights are does matter in how much better the model gets and also matter in regards to the cost of training / inference. Model design is a big set of trade offs and I see MoE as a useful tool that will survive in the trade off space.

>reasoning is left on the table

Even so, if there was 2 models with an equivalent amount of reasoning ability and priced the same would you rather pay for the one with narrow knowledge or wider knowledge.

>because of the paucity of modern dense models at the near frontier,

You don't need to be at the frontier to benefit from MoE. Even open source models that are behind the frontier, benefit from being able to host experts on different machines, and scale individual, commonly used experts separately from each other. On the other end with small models you are probably resource constrained so you want to maximize the tokens generated per second. This makes going for purely dense models niche like you are saying.

[reinitctxoffset]

I think we're fundamentally reading from the same sheet of music but drawing different conclusions.

Mixture models have compute advantages at training time, everyone agrees about that, that was the original rationale (popularized at the time with `mixtral-8x7B` among others). This seems to be likely to remain an economically relevant strategy for (especially) pretrain: in a training setting you have already paid for fast interconnect at scale, a dense architecture doesn't buy you anything in a big pretrain and it costs you a lot of traffic and to a lesser degree batch size under the roofline. The heavy, FLOPs intense, interconnect intense parts of training run benefit enormously from MoE: I don't dispute that though I suspect we are well into convergence on the target precision (4) and the target format (NVFP4 or similar). At some point the whole Internet is in the pretrain at the terminal generalizing precision, the pretrains of the various labs start to look a lot alike, and the sauce remains in the later parts of training along with the proprietary data sets and what not. Frankly all the labs would benefit from standardizing the Common Crawl recoverable pretrain to greater or lesser degree, it would lower everyone's costs without changing the competitive landscape much. But that's my prediction/opinion, that's why I said "suspect".

The evidence is suggestive if not fully conclusive that mixture models are strictly losing in most regimes during inference: the exemplar of Qwen3.6-27B (which outperforms Alibab's own mixture model at ~ ten times the size) is very suggestive, and it's not the only argument for this. Because most/all modern MoE requires the activations of the previous layer before routing the subsequent experts, you are pretty much paying for the HBMe3 or GDDR7 to hold the whole thing even though some small fraction of it is under your roofline on any given token or draft verification.

This is grossly wasteful under all trajectories (even parity at N parameters between dense and MoE, which we have evidence is off by 1-2 orders of magnitude): in a "local LLM" setting (from bedroom to regional office, anything other than an NVL72 or Ironwood rack) you are sharply constrained by both total available accelerator DRAM and accelerator memory bandwidth: you are probably not getting under your roofline even with pretty slow tensor units (e.g. GB10). This use case matters and looks like it's going to matter more and more over time (the GB10 in particular is going into about a gigaton of RTX Spark laptops next year). In a datacenter setting, you're paying for extreme interconnect (training class hardware setups basically) for pure forward pass that wouldn't otherwise need training optimized gear. Even multi-trillion parameter dense models can fit in 8x or 16x RTX 6000 Pro style setups (hell, there's a DGX branded one) and with all of the geometry, tiling, scheduling, and interconnect needs mapped out up front, the design space is really forgiving on all manner of tensor parallelism, pipelining, KV cache sharding, it's a very friendly constraint space up to like, 3-5T parameters. MoE at inference time works for two groups of people: people who are willing to page experts in per token/draft batch in local LLM settings (not probably ever going to be mainstream, people really dislike that level of slow), and the vendors of extreme performance interconnect i.e. vendors selling training-class equipment as necessary for inference. And you pay in so many other ways: grouped GEMM is no one's idea of a good time, the kernels are fiendishly difficult, therefore they are not abundant, it's no fun.

The path forward here is non-obvious, and I don't claim to have it all figured out. But since you seem interested enough to carry the conversation past the pleasantries, a more substantial version of my thoughts on the matter can be found at: https://cdn.s4.gl/preprints/pairwise-interlock.pdf

[int_19h]

MoE is just activating fewer weights per token than the whole model. It will continue to make sense for as long as compute is more expensive than memory (at scale).

[dominotw]

even meta that sucks at doing anything is releasing frontier models. making an top ai is easier than making twitter clone( threads) if you have enough money.

[password54321]

I mean the problem with Threads was lack of user engagement. The same could possibly still be said about their models.

[dominotw]

yes ofcourse. But engagement needs strategy and execution.