[kragen]

It matters if studies like this matter, that is, it matters to people who are interested in what has currently happened rather than what might happen in the future. 6 months of LLM progress keeps not looking like what I expected.

On the other hand, if you're content with your pre-existing predictions about what would happen, which I think is actually a reasonable position, there's no reason to read the paper.

[fzzzy]

Is progress faster or slower than you expected?

[kragen]

An astounding amount of both.

[fzzzy]

That makes sense. Progress does seem very lumpy. Commercialization and secrets means that what we see publicly may not be state of the art. And I certainly never thought that Zuckerberg would be the one providing open model weights. It's a strange new world.