[sandcat_]

Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a new model drops.

Instead, it feels like a more appropriate benchmark for the original purpose would be to come up with new, novel problems each time, and compare across all models (including previous ones).

Throwing my hat in the ring: Generate a pelican shaped crossword where all the clues are related to bicycles.

(Haiku 4.5: https://imgur.com/a/N112Nxo, I'm trying some others but it's very slow! Opus has been at it for about 20 minutes.)

[phreeza]

I think a lot of LLM progress in the last 2 years or so is really from densifying the training data to niches like this and not so much from general problem solving capability. I guess the hope is that the deluge of super specific training data will pay off through better overall competence. cf https://www.dwarkesh.com/p/the-sample-efficiency-black-hole

[qq66]

I do have a private benchmark I use for that. I have a repository with an in-progress codebase and a very large planning document, I tell the new models to simply finish everything in the planning document, and I never push those changes. All of the recent models have essentially gotten it perfect which means that the models are all equivalently good for my level of needs.

[londons_explore]

I have noticed that codebases I have used an AI on seem to perform better in future AI models than ones I have not.

This makes me think AI companies are using chat history to train the next model.

[tln]

AI generated code is usually full of comments, human code is usually less commented.

It may be that AI performs better on such code.

[TomatoCo]

I've mostly been running LLMs on my own hardware so the phrase "been at it for about 20 minutes" gave me pause. My first instinct was to ask "but on what hardware?" but I suppose one advantage of closed models is that there's a somewhat-consistent cognitive-effort-to-time ratio.

[sandcat_]

Fair, I don’t know if this helps but it was Opus 4.8 (high effort) via Claude Desktop, but through my org’s LLM gateway so hard to compare really. But it’s usually pretty quick.

(don’t tell my boss.)

[amelius]

How they will approach it eventually will probably be to let a diffusion style model render the image, then run some algorithm to convert that to svg.