[simonw]

This is fantastic

I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations.

Catching a lab cheating specifically on my one dumb benchmark would be really funny.

Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than anything I was considering.

His conclusion:

> Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid.

[lukev]

What if they’re not pelicanmaxxing, but svgmaxxxing in general?

Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.

Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

[qq66]

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

[sandcat_]

Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a new model drops.

Instead, it feels like a more appropriate benchmark for the original purpose would be to come up with new, novel problems each time, and compare across all models (including previous ones).

Throwing my hat in the ring: Generate a pelican shaped crossword where all the clues are related to bicycles.

(Haiku 4.5: https://imgur.com/a/N112Nxo, I'm trying some others but it's very slow! Opus has been at it for about 20 minutes.)

[phreeza]

I think a lot of LLM progress in the last 2 years or so is really from densifying the training data to niches like this and not so much from general problem solving capability. I guess the hope is that the deluge of super specific training data will pay off through better overall competence. cf https://www.dwarkesh.com/p/the-sample-efficiency-black-hole

[qq66]

I do have a private benchmark I use for that. I have a repository with an in-progress codebase and a very large planning document, I tell the new models to simply finish everything in the planning document, and I never push those changes. All of the recent models have essentially gotten it perfect which means that the models are all equivalently good for my level of needs.

[londons_explore]

I have noticed that codebases I have used an AI on seem to perform better in future AI models than ones I have not.

This makes me think AI companies are using chat history to train the next model.

[tln]

AI generated code is usually full of comments, human code is usually less commented.

It may be that AI performs better on such code.

[TomatoCo]

I've mostly been running LLMs on my own hardware so the phrase "been at it for about 20 minutes" gave me pause. My first instinct was to ask "but on what hardware?" but I suppose one advantage of closed models is that there's a somewhat-consistent cognitive-effort-to-time ratio.

[sandcat_]

Fair, I don’t know if this helps but it was Opus 4.8 (high effort) via Claude Desktop, but through my org’s LLM gateway so hard to compare really. But it’s usually pretty quick.

(don’t tell my boss.)

[amelius]

How they will approach it eventually will probably be to let a diffusion style model render the image, then run some algorithm to convert that to svg.

[Balgair]

https://www.youtube.com/watch?v=jgYYOUC10aM

reminds me of this Key and Peele skit

[ralusek]

Alternatively Mitchell and Webb https://www.youtube.com/watch?v=_pDTiFkXgEE

[ryukoposting]

How useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good.

AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.

[derektank]

Just today I asked Claude Fable to take an SVG map and add a circle with a 150 mile radius around a specific city, cutting off the circle at the edge of certain boundaries. It came back a minute later with more or less exactly what I wanted, saving me maybe 5 to 10 minutes of photoshop time. It’s almost certainly not perfect, but I didn’t need it to be perfect, just a legible representation for a group of 20ish people.

[somenameforme]

OTOH I had Fable laying out a 3d broadcast room, and as part of the assets it sourced, it grabbed a camera. Makes sense. There were a lot of problems with the scene - backwards facing props and clipping everywhere. But the true coup de grace was a mansized, more like a giant sized, 35mm camera pointing at everything. I was seriously confused for a minute until I realized what had happened.

[cmckn]

I used Claude to plan some landscaping, and it gave me SVG diagrams of flower beds showing where to place plants and their approximate mature size. It was pretty useful.

[zahlman]

Even if the models don't break through any particular "uglier than sin" barrier, with a bit more work, presumably the SVGs could become importable into an editor that would let a human apply taste and discretion. Seems to me like a heck of a head-start.

As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.

[fiddlerwoaroof]

AI can generate a fairly satisfactory SVG for a favicon now (programmer art quality at least).

[Kamshak]

Great for charts, illustrations etc, can do much more with SVG than pretty pictures. Good spatial understanding in an SVG can be really nice for model to have

[SirHackalot]

I used it to generate some icons for my app recently, I found it a pleasantly useful feature already in its current state. Before this I would’ve had to close the code editor, open something like Illustrator, use something like a tablet (or struggle with the mouse and keyboard), and export the result. For something like a generic icon for a button, or a simple geometric figure, this always felt too tedious. On the other hand, my apps looked visually impressive before the AI and people valued my artistic ability more. So there’s that… Not every software engineer also had an eye for design. With AI generation, they can bridge that gap.

[amarant]

Vibecoding a SVG based metroidvania as we speak! This is gonna be lit!

[joshmoody24]

Would it be useful if the models were actually good at it?

[throw10920]

> benchmarkmaxxing on a weightlifting competition

If you stick to the benchpress, it's just "benchmaxxing".

[isoprophlex]

I'm about ready for maxxmaxxing, where there's just maximally more of everything all the time

[Sharlin]

I’m waiting for the inevitable counterreaction to all this maxxing business, dubbed "minmaxxing"

[Paul_Clayton]

In D&D, min-maxing (hyphen and one x) is already a used term for character optimization, minimizing undesirables and maximizing desirables (I guess).

Minimizing peaks is probably not a good strategy generally. Even low peaks may have some benefit ("if you know your problem is in this domain and performance is critical, this tool offers a 3% advantage").

Minimizing troughs may be more attractive. People generally seem to react more strongly to negatives and competitors can devise benchmarks which emphasize one's troughs.

While a universal expert would be convenient and broad knowledge aids some forms of creativity, specialization has substantial advantages.

Trading max performance (primary metric of concern) for some improvement in a secondary metric often makes sense and that could be an interpretation of "minmaxxing", reducing the over-emphasis of a single metric which would otherwise be maximized.

[OutOfHere]

The fear is that the SVGmaxxing is limited to "X doing Y". If such 'template maxxing' exists, it will break for other templates, e.g. "X not doing Y", "X and Y doing Z", "X doing Y doing Z", etc.

[nl]

I use LLMs for 3D CAD design in OpenSCAD.

There seems to be a very strong correlation between models that are good at SVG and models that are good at 3D CAD.

Anecdote I know, but there does seem to be generalization going on here.

[somenameforme]

Which models have you found good for working with 3d stuff?

[nl]

ChatGPT 5.5 and Sol 5.6, Fable are good.

I haven't really tested Opus 4.8, but 4.7 wasn't nearly as good as ChatGPT 5.5.

[zarzavat]

If a model can improve at drawing "X doing Y" and that prompt wasn't in the training set then it means it has improved its internal mapping from text-to-spatial-to-text.

[OutOfHere]

The improvement would be limited to the template "X doing Y". It would not be a general improvement unless hundreds or thousands of diverse templates were used.

[tempfile]

Ugh, they're just goodmaxxing.

[seebeen]

I love this comment.

[sysguest]

well that holds IF svgmaxxing is 100% "code-writing-maxxing"

...which.. hmm I dunno if they are same or not

[tsimionescu]

No, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding).

That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.

[runarberg]

If people need to generate good-ish SVG why not simply use a specialized model for a much better result and for far cheaper and quicker?

Why do LLMs need to be able to do this as well, but worse, slower and more expensive?

[yorwba]

Yeah, if you want to generate an aesthetically pleasing SVG, you'd be better off asking a pixel-based image-generation model for "vector art" and then deconstructing it into an equivalent SVG with something like LayerPeeler. https://layerpeeler.github.io/

[nl]

For example I do SVG diagrams to explain architecture and specific data flows in a multi-tier app.

LLMs do a great job because they understand both code and SVGs well.

Edit:

An example for a synth I'm buulding: https://imgur.com/a/U694Ek7

The irony of having to post it as a PNG isn't lost on me...

[exhaze]

Agree directionally - even back in sonnet 3.5 days, I was helping some friends by showing them how to create intermediate representations for SVG building blocks mapping to parametrizable functions that can be used to make interactive SVG-rendered visualizations for various medical needs.

[wasabi991011]

I don't see why that's true. LLMs don't have to only be good at code-writing.

[Theodores]

As a human SVG-maxxer, I doubt there is the training data out there for true SVG maxxing.

SVG has its neat SVGO tool that optimises SVG code, this is very clever but it won't spot easy wins such as when a group of elements can be mirrored, or an element re-used with a transform. Neither will it use the full smorgasbord of features. SVGO is not AI, however, it does a fab job of taking bloated files from Adobe Illustrator and getting something good to work with.

The bar for SVG is really low, mostly just paths, compressed and not human readable. SVG should be a human readable format, so 'circle radius 10' rather than two hundred points at six decimal places to draw the same circle.

You have to RTFM to do cool things with SVG and there is a lack of appreciation of the format amongst developers and designers.

[timClicks]

If they're optimizing for SVG generation, then that's an excellent outcome in my opinion. Vector images shouldn't be "pretty niche".

[21asdffdsa12]

And SVG is fantastic to draw easter eggs or hidden messages. Just a insult to the top of the 4 in 404.

[knollimar]

Please it's almost my primary bench for drafting understanding.

I'm almost about to post that xkcd 810

[rtpg]

> Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

I don't understand the truth to this point, and there's a clear difficulty in establishing structure, a thing that you don't need to actually do with raster image gen, right?

Like if you say "please give me 9 circles" you expect circles in the image and not just a bunch of pixel that are vaguely circle-like, right?

[simonw]

Gemini have absolutely been SVGmaxxing. They've openly talked about it.

[brikym]

Then it's great. A year ago I couldn't get any AI to draw a simple company logo in SVG or even convert from raster. No doubt the Pelicans put pressure on the labs to fix the awful SVG situation. Now I can even make a decent Peli in 3D.

[exhaze]

Not an expert on this, so won’t speculate regarding what traits would create robustness specifically attributed to SVG visual representation capability, but felt you might find this paper on reasoning models trained on physical world video data becoming better at general reasoning interesting:

https://arxiv.org/abs/2210.05359

[netsec_burn]

Addressed in the article, in case you're curious.

[Dylan16807]

Addressed by three inconclusive sentences. When someone goes into deeper detail on the topic, we shouldn't assume they failed to read the article just because they said "what if" instead of mentioning those three sentences.

[lukev]

Well, it’s mentioned as a limitation of the analysis, very much not ruled out (or in.)

That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)

[salomonk_mur]

Generating images not by pixels but by code and instructions is niche to you?

[kelvinjps10]

SVG are really useful you can create images that don't have the AI look

[Eggpants]

I've been pretty happy with LLM svg based data plots I've asked, including log scaled axises and histograms. Definitely a first world problem of course.

[mgrandl]

But did they create those plots “by hand”, or did they use one of million open source libraries that create SVG plots? In my experience even SOTA models suck at generating SVG graphics like logos.

[6thbit]

I think svg is a balanced test because of the level of indirection and the required 'conceptualization' of physical elements then expressed through code.

[kaliqt]

Funnily enough, not that niche, because I have tried many times to do it as part of a wider project.

[nbardy]

Then they’re legitimately getting better at svg which is a valuable tool.

[charcircuit]

I agree, other formats, both textual and binary should be tested.

[meowface]

They arguably might be, but I kind of want them to be!

[Melatonic]

Probably the case but not a bad one to "max"

[hahahaa]

svgmaxxing, please do it!

https://xkcd.com/810/

[delichon]

> What if they’re not pelicanmaxxing, but svgmaxxxing in general?

Mission fucking accomplished. https://xkcd.com/810/

[eob]

Simon I hope from this day hence, your bio always includes:

"Simon Willison, among other things, is an advocate for the inclusion of pelican geometry in LLM training datasets."

[exhaze]

“Today, the NASDAQ fell by 18% moments after Simon Willison published his latest blog evaluating the Legend 7.4 model that rendered an animated SVG of a pelican on a bicycle and the pelican fell off”

[shawabawa3]

When asked for comment an anthropic spokesperson said "it's very unusual for the pelican to fall off. They are built to very rigorous SVG engineering standards"

[julianz]

I asked Gemini 3.6 Flash for a pūkeko on a unicycle yesterday and the results were very stylish. The thinking notes were cool too, making sure it got all the high points of what it means to be a pūkeko (distinctive swamp hen found in NZ and Australia for anyone not familiar) and then figuring out how the legs would work with the pedals.

[packageman]

I worked with image analytics years ago. In 2024 and cynical about LLM image capability, I tested different models to create "Two female octopi in a bar each having their own drink. One is wearing a yellow hat. The other is wearing a red hat. They have no human features." It went badly I think because the training visual training data associated with the word "female" was overwhelmingly biased in quantity and obviated the 'no human features' instruction even after iterated instructions to remove.

So I tried "Make an image of a Djibouti cab with a camel sitting in the passenger seat. Give the camel no human anatomical features." Still bad.

Two years later the graphics and perspectives of the tools are so much better. But that isn't the big story. What really stands out is the models are no longer adding human female anatomies to camels and octopi. The gates and filtering based on instructions have matured enough so that the pelican/bicycle deductive reasoning puzzle is less problematic. But until an LLM anticipates something like impressionism from Parisian artists rebelling against the rules of the French Academy, continue to reserve a place for human artistry.

tl;dr simonw, your spot-check nailed it just as well as any extensive methodology. LLM's no longer need to cheat this part of the test (better to hack the question than the tool).

[Sharlin]

Back in 2024 we didn’t really yet have image models that used bigger language models as their text encoders, so prompt understanding was still rather hit and miss, especially with negations involved.

[gopalv]

> Catching a lab cheating specifically on my one dumb benchmark would be really funny.

Similar thing happened when TPC came up with SQL benchmarks.

If you're not good at TPC, your engineering team is no good.

If you're good at TPC, then (as a customer) we will actually include you in a bake-off benchmark for our specific problem.

Winning on it is the price of admittance into the game, especially in a crowded market.

But how narrowly you benchmarket matters, you can't just hard-code that specific scenario & not fix anything adjacent while you're at it.

For example when it comes to GPUs, the "Quack3" (sic) benchmark on ATI cards comes to mind.

[ryandrake]

Yea, I worked for a competitor to ATI back in the day and we were definitely Quakemaxxing. We were not so unethical as to try to detect the .EXE name so to put the GPU into a "cheating" mode only when that benchmark was run, but we did run Quake 3 pretty much constantly while trying to eke out a few more FPS...

[exhaze]

Wasn’t that almost the norm for both new OS drivers and subsequently GPU SoC firmware of ‘gamer cards’ to literally optimize for the latest AAA titles? I feel in that case, the interests aligned - in a world where I had a chance to actually play Crysis, I wouldn’t care why the frame rate was decent, no?

[exhaze]

What if they’re just unsuccessful at it and something about a model grokking how to create a realistic visual representation of a pelican on a bicycle ends being key to the next OOM of capability unhobbling.

And since we have established this silly routine once, I must keep going and ask - yes, you have quite a collection of bona fide pelicans you’ve seen and photographed.

Have you physically seen all the other animals you’ve evaluated as well?

I was born in Soviet Russia, so trust but verify and if still around, maybe there pelican make LLM draw Simon ride bicycle and notice Python code improve.

[gilleain]

Perhaps also vary the bird? Wikipedia tells me pelicans are in the order _Pelecaniformes_ so shoebills or herons might do.

[docheinestages]

I think a more fundamental test is SVG art creation in general. Perhaps a pipeline to take any image, caption it, ask the LLM for an SVG, rasterize to an image, and finally either use a deterministic visual similarity check or ask another LLM to be the judge and score how close the SVG is to the original image.

[zahlman]

Fidelity to the original is definitely not how humans would measure "art" in this context.

[docheinestages]

True, maybe we can call the generated SVG something else than art.

[JadoJodo]

It's a shame you started with pelicans on a bicycle and not snakes on a plane.

[usef-]

It's better to be original

[cyberax]

> I've been casually spot-checking other animals in other vehicles

Snakes on a plane, weasels on a diesel, spiders on a glider, baboons on a balloon, goats on a boat.

[zkmon]

Why is it hard to come up with tests that closely resemble real-world usecases where the models are being used?

[simonw]

It's not that hard - there are plenty of those.

They're mostly not very funny though.

[philipwhiuk]

Would it be any more applicable?

We see the same problem in database benchmarking.

The good thing about the deliberately-non-real world pelican case is that it gives a general impression of how much the model is improving because it's not likely that it's being specifically targeted at it, rather than a 'real world case' which might have been specifically optimised for.

[DonHopkins]

A pelican on a bicycle is like a canary in a coal mine.

And a man without religion is like a fish without a bicycle.

[Melatonic]

It's all fun and games until that Pelican becomes self aware and asks themselves - why am I riding this bicycle ? I could be creating a Terminator 2 esque dystopian future myself.

And that's how we end up with PelicanSkynet

[graemep]

I am not sure being better at pelicans on bicycles would be proof of cheating. Could all the commentary on the web on pelican on bicycle SVGs not influence training without anything being done deliberately?