[galaxytachyon]

How good is it at scaling? And will it still retain the emergent capabilities of the huge transformer LLMs?

Isn't this basically the bitter lesson again? Making small improvements work but in long term it won't give the same impressive result?

[coldtea]

So? Would you rather we didn't make small improvements?

If we could just make big improvements we would.