[impossiblefork]

I think the Kimi thing is super cool, especially that they have so many RNN/linear attention layers (3x more than they have full attention). I haven't yet tried it though. It seems like it would be extremely reasonable for long context tasks and I guess this fits the times.

I suspect that the reason it has so many parameters is the same reason that compute optimal xLSTMs have some many parameters, and the success of this model makes me a bit unhappy that we haven't gotten an xLSTM-style model of huge size developed in Europe.

Obviously these guys are very pragmatic, they're probably not committed to anything other than what works on their internal evaluations, so they still have ordinary attention layers in the model and so on, and one can't be guaranteed that the people who come up with a good model then do the engineering in an ideal way, but I still think the success of Kimi shows what could have been if we had enough big supercomputers for LLM training and made them available to the right people-- because this is basically Hochreiter's thing. It's RNNs, or well, mostly RNNs.

[georgeburdell]

Wasn't one of the original intents of the transformer architecture to get rid of RNN because they cannot be parallelized? Not an expert, just read a few papers several years ago.

[impossiblefork]

Yes, although then people came up with reasonable ways of parallelizing RNNs entirely or in part anyway. There's also the need for lookups in transformers. Queries still need to be multiplied with old keys, all of them, so RNNs don't have to be totally incompatible with parallelism. It's just that you can't have a matrix inside them like in the linear RNN h_t = Ah_t + Bx_t that you have to multiply together step after step. If A is an input-dependent scalar or something you can multiply together easily you're fine.

mLSTM layers are completely parallel (the update rule for the cell state is C_t = f_t C_{t-1} + i_t x_t where f_t and i_t are gates that can be computed from x_t alone, x_t is the input at time t and this means that you can compute F_s = \prod_{s<t} f_s as fast as a cumsum and an exponential and then compute C_t = \sum_{s<t} F_s i_t x_t, again as a cumsum, so it's as good as if though it were parallel). I think the sLSTM layers that are the other component of the xLSTM have something else like this and presumably there's some trick also to training the Kimi "delta attention" RNN.

I'm not sure whether this is hardware or optimization dependent to some degree, but I get the impression that good custom kernels are an important part of this kind of thing.

[michalsustr]

I talked with Hochreiter about 1.5y ago they should go for it and properly scale xLSTM to LLMs, not just playing around with few B params for experiments. They felt like they are too late to the game to get investment and traction. So much for European ambitions I guess...

[impossiblefork]

Yes, but how would it have been if the EU governments just piled on, subsidized whoever had reasonable qualifications who wanted to try, straight-up funded accelerator development so that the chips don't just become investment in US AI companies, straight up built the supercomputers for training, straight up offered access to groups that had previously had reasonable success in training some model.

I don't think it would even be that expensive. I think with EU chips we could do it for 20 billion EUR + 5 billion EUR per year. After all, an H200 doesn't cost 40k to manufacture, probably one-tenth, and if we are making the chips we can get the money circulating just as the Americans have.

Add in some state effort to create suitable training data-- maybe encourage academics to create some things that are AI-friendly, and let all the people trying have that. There would hardly be anyone in the game other than us if we went for this package.

We already have VSORA. Maybe Euclyd is ready soon too. If enough are ordered we have 85% of the inference need gone, so that whatever accelerators are freed can be repurposed for supercomputing and training? Maybe we could have some organized effort to make many of the VSORA and Euclyd chips so that the chips we need for training are freed up while we wait for our own training chips.

Edit: So in the end I think the thing we need to do is to order enough VSORA Jotunn8 cards and enough Euclyd cards that we flood the inference market so much that the price of H200eds etc. drops and we can buy them second hand and assemble huge supercomputers out of them. This may require some diplomatic effort from the commission, something like 20B EUR almost immediately and some coordination, but it's easy, it's doable and it requires only being a little bit decisive. Since these cards are faster less memory will be tied up in serving the models, so it'd have a positive effect on memory prices too, which in turn would have a positive effect on European supercomputer centers, who are currently of course constrained by high memory prices.

[eru]

What's their track record for other state sponsored high tech work?

I remember a few high profile failures; but perhaps on average these things work out?

[impossiblefork]

I am actually not quite sure.

Here in the EU Rhea 1 was delayed and thus maybe a bit of a failure, but Rhea 2 might be good and seems to be on time. OpenChip seems to be on time too. The university work, people like Hochreiter etc. is clearly good.

What are the high-profile failures?

[miohtama]

The EU spends EUR 300B yearly on agricultural subsidies and social programs.

[impossiblefork]

Mm.

Still, maybe less money could be enough.

If Euclyd's chip costs 40k per chip and we want to substitute all worldwide H200 inference capacity, so let's say 700,000 H200eds and Euclyd are right about how good their chip is, then we'd need 28,000 Euclyd cards, this is only 1.120 billion.

So maybe even 1 billion for VSORA chips this year and then two billion for VSORA and Euclyd chips next year, with the split based on how well the chips perform and some money for the lagging firm so they can keep designing things too.

[alex7o]

Is it an RNN or a state space one, like qwen or there was an arch called mamba if I remember correctly. Or it might be closer to RWKV.

Edit: I checked and it is similar to linear delta net uses by qeen.