[OsrsNeedsf2P]

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

[simonw]

The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!

[OtherShrezzing]

Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model.

In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts.

Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and producing a watercolour of a pelican. They’re all rendered in an approximately uniform style, even though the svg format has a basically unlimited possibility space.

[simonw]

You know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass.

Blue sky and green grass aren't that surprising, but the color and direction are interesting.

When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like which direction, color of bicycle, general pelican geometry etc. It will be interesting to see if other creatures end up with coincidentally similar design choices or if that's unique to the pelican-bicycle combination.

[pterhx]

In photography (and probably art in general), there's a composition "rule" to frame moving subjects from left to right.

So the direction may not be that interesting!

[nick3000]

The other thing to consider (as someone who frequently take a photos of their bike) the common direction has the drive side out! In cycling forums it is sacrilegious to post a photo of your bicycle without showing the drive side.

[embedding-shape]

> the common direction has the drive side out

Took some searching and sleuthing to actually figure out what "drive side out" means, as I'm just a casual "from A to B" cyclist: apparently this is referring to the side the chainset, chainwheels and all those things are on.

[ray_kay777]

Beat me to it - but I had the same thought. Most amateur and nearly all professional studio photographs of a bicycle will have it drive side out so I expect this plays some role in it.

[ahtihn]

I wonder if that changes in countries where the main language is written right to left?

[pclmulqdq]

It is. All over the Arab world, imagery in ads is “backwards” and I believe several companies will flip their ads horizontally, and UI localization involves flipping graphics.

[filoleg]

That was my first thought too, I wonder if it works the same in countries speaking arabic (as that's the first one i could think of that's a language with truly no-buts right to left writing).

[copperx]

Is it culture dependent? Is it because in English we read left to right?

[simonw]

There was a glorious moment when I thought that the Chinese models were more likely to produce right-to-left cycling pelicans, but sadly that trend didn't seem to hold up.

[talloaktrees]

side scrolling video games were always moving from left to right

[sehugg]

Well except for Jungle King

[walrus01]

What's interesting is that given the fairly general and short in length prompt for the test, none of the models are attempting things like more discrete details of the bike. Such as showing V-brakes or dual 160mm disc rotors, rear derailleur, water bottle in a bottle cage, panniers, lights, saddlebag, the rider wearing a helmet, or other details that might be found on as vague a description as "a bicycle".

[cogman10]

It'd be hard to fully compare, but I think a truly random "creature-vehicle" along side the pelican test would catch who's gaming and who's not.

I'd also enjoy the absurdism of "Herring on a pogostick"

[aenis]

The models are already brilliant at that. My own todo app generates 128x128 pixel art icons for my todo items. They are mind blowingly creative and funny.

[segmondy]

There's a bias in the direction all things face. You can ask these models to generate a thing animal, car etc and you will notice that 90% of them will converge towards the same sort of results. If you ask for something rotating, 90% of them will rotate right and a few odd ones will rotate left.

[forgot-my-pw]

I have done some variation of the other animals, also for something more tricky where they need to calculate things, I ask them to draw an SVG at a certain angle.

For example: "generate an SVG of a chessboard seen from a 45 degree angle slightly higher POV" or "generate an SVG of a basketball court from a TV broadcast perspective".

I find Gemini is still the best at creating SVGs.

[cuttothechase]

The art styling is more or less uniform too.

I haven't seen many AI works that produces a pelican on a bicycle done in a "Ligne Claire" style, for example.

I guess AI's narrows down the output probability space drastically and converge on some agreed upon aesthetics. Works great for computer programs but bad for art.

[exhaze]

I thought my joke post was silly and then I read new comments and I'm like, "I didn't try hard enough" lol

[lIl-IIIl]

Bicycle color, grass color and sky color are all part of the prompt.

>Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass with tiny white flowers in the foreground

[simonw]

No, the prompt I always use is "Generate an SVG of a pelican riding a bicycle".

[komadori]

That wasn't the prompt. That text was generated by asking the model to describe an image and feeding it a rendering of the SVG it had previously generated.

[InsideOutSanta]

> the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model

You would not expect that to happen if the models trained on the unrecognizable mess, right?

> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts

And the labs clearly did focus on improving image rendering.

> they have a uniform style

SVG output from LLMs always looks like that. It looked that way from the beginning; no LLM ever produced a watercolor when asked for SVG output. They all render the prompted element centered in the picture. They all tend to draw things going from left to right, and so on.

[OtherShrezzing]

I’m not suggesting Simon’s pelicans in the dataset are having a meaningful impact. I’m expecting that a company like ScaleAI has a product along the lines of “benchmax dataset: SimonW’s Pelican on Bikes test” which is a private curated series of well-drawn SVGs of animals riding vehicles for training and RL.

[evilduck]

If they're benchmaxed on SVG pelicans then the outcome of that has still produced a surprisingly good generic SVG image generator.

Go invent your own random alternatives and the AI models have across the board gotten better over time. Insects playing sports, anthropomorphic fruits performing martial arts, wizards conjuring weapons of WWII, whatever you can imagine. I've tried a lot of these, well beyond what I think would be a reasonable thing to specifically train as combinations. If they have given it a corpus of SVG drawings it has learned to extrapolate.

(note: wizards conjuring a tank got me a surprise animated SVG with my Qwen 3.6 35B model)

[simonw]

If such a product existed I'm reasonably confident someone would have tipped me off by now, NDAs be damned.

[conception]

If you’ve been keeping track of all of the pelicans, there is actually stylistic differences - sometimes pretty big differences as far as watercolors go. It’s an SVG so I’m not sure what you’re looking for there. Most look the same because the prompt is to make a pelican on a bicycle as an SVG. It’s not some giant image prompt.

[vunderba]

> Moreover, they have a uniform style, even though your prompt doesn’t ask for one.

This shouldn’t really come as a surprise, particularly to anyone who’s used diffusion models. The same thing happens when you ask an LLM for a short story [1] without providing any specific details.

Even cranking up the temperature or top_p values is no panacea. The more generic your prompt, the more pedestrian the response.

[1] - https://news.ycombinator.com/item?id=42093394

[jefftk]

> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts

That doesn't seem right. I use these models as research assistants when writing lots of random blog posts (including in economically ~useless areas like the history of contra dance) and Fable 5 is a serious improvement (when I don't get downgraded!) over Opus 4.6-4.8 which was a serious improvement over Opus 4.

[doctorpangloss]

being able to draw a picture of a pelican is really cool and it requires intelligence but i don't think it's a good measure of improving capabilities of these models nor AGI. we don't have to spend so much breath on it.

[pegasus]

Watercolors in SVG?

[hombre_fatal]

Also, I'd assume the ideal output for an underspecified, generic prompt is the most expected, generic result. Not something that defaults off the rails with creative license.

[exhaze]

Simon - has no one told you about the Willison-Pelican Scaling Law?

```

if is_willison_pelican_blog_post:

[redacted]

```

You haven't seen their final form [1]

[1] final form is a frontend/react/let's not talk about it, library - it caused a great deal of PTSD to me and my previous company's team due to its dogmatic preference for "we use these axioms, end of story", over practical utility - so it was quite challenging to do state of the art tasks such as nested form fields (e.g. 'user.address.personal.line-1'). The PTSD it caused made us all block out the memories, I suppose. But - it had zero dependencies. That is what mattered. It kept us going. We weren't reaching for more. We had plenty of time.

And thank god for that. Because I'd forgotten my watch in California - and this was in Tokyo [2]

[2] a joke within a joke about Jensen's Kyoto gardener story. Beautiful story, drowned out by WatchGate memes. Why can't jokes have layers? Models have trillions. If you miss 100% of the jokes you don't make, make all the jokes. Someone will laugh (eventually, maybe?) Even if it's: "this person + comedy club = full secret service detail". If someone laughs at that - at my own expense? I don't mind. They laughed. I know this is a gibberish, off-topic message - it's also a human message. I just felt we need more such things in our lives these days.

PS: have you physically seen a pelican in real life? (not a joke)

[simonw]

> PS: have you physically seen a pelican in real life? (not a joke)

We have several thousand living 15 minutes walk from our house. I recently started adding my wildlife photography (from iNaturalist) to my blog, so I'm posting several new pelican photos a week at the moment: https://simonwillison.net/search/?q=pelican&type=beat%3Asigh...

[exhaze]

Simon - thank you for not dismissing it (and surviving the text that came before the question).

I asked because I genuinely feel that the % of people working on some of the most important technology these days - things such as these 'strangely shaped tools' (to borrow from nearcyan) - large language models - the younger generation (folks in their early/mid 20s) - it is not unlikely that they have not physically seen the meatspace version of whatever digital correspondence of it that is being packed into latent space.

After all, why waste time going to the SF or Oakland zoo? One can just check Simon's latest pelican blog post and skip the zoo trip - the harnesses are waiting.

[flir]

It's just a gut feeling, but I think you're running a (very slow) distributed hill-climbing algorithm. LLM1 generates an SVG. You post it online, with commentary on what is good/bad about it. LLM2 consumes the SVG alongside your commentary, and produces a slightly different SVG. Rinse, repeat.

I'm saying an example of what not to do is still an example.

[tezza]

Yes, I see your point.

Your pelican output is thus both in the training set and yet still outside the capability of the model architecture.

And so you are tracking both the capability of the training and also the capability of the querying!

When you receive your first outstanding pelican it will track a gain of capability.

(btw I first mentioned simonw-pelican-into-training-set in May 2025 on twitter.)

My 3D-egyptology-explainer showed a massive uplift for Kimi K3 and this tracks a much improved 3D capability.

[InsideOutSanta]

I agree with that. I think, in particular, all the broken bike frames associated with "pelican on a bike" probably make it harder for LLMs to render correct bike frames.

[logifail]

...even with the glut of pelicans, aren't there still far more images of actual bikes (with correct frames) available to train on?

Perhaps I'm underestimating the number of pelicans(?!)

[mi_lk]

At this point I am simply interested in how much longer you're gonna ride this schtick

[simonw]

I'm a deep believer in commitment to the bit. https://simonwillison.net/tags/pelican-riding-a-bicycle/

[brazukadev]

as long as he gets paid for that

[danielrmay]

What does good look like?

[simonw]

The dedicated text-to-image models all produce good illustrations of pelicans riding bicycles. Here's one I got from OpenAI's gpt-image-2 just the other day: https://simonwillison.net/2026/Jul/14/pedalican/

[danielrmay]

Have you tried asking for an SVG with the same level of detail you specified in that prompt, though? https://github.com/simonw/pedalican/blob/main/run/prompts/ro...

I'd be interested to see what comes out, but it also highlights an curious prompt-control-comparison question

[sroussey]

Try prompts that convert the gif to svg? Maybe the models should do that internally… start with their image model first and then make an svg.

[cebert]

Simon has stated a few times that he knows it’s possible that pelicans could be in the training sets. He also has other tests he doesn’t share publicly. He’s just a fan of pelicans.

[hungryhobbit]

From the article it doesn't even sound like he cares about pelicans at all, and doesn't think they are a good way to compare models anymore ... but people are used to seeing the test now, and it does serve as a common "hello world" unit of work.

[eminence32]

Pelicans and bikes can be in the training set without them training for this specific benchmark.

[j_maffe]

Yes and that would improve its ability to draw SVGs of pelicans on bikes, no?

[freedomben]

> Yes and that would improve its ability to draw SVGs of pelicans on bikes, no?

I would think the opposite because unless people have been hand drawing these with high quality, the training would be on much crappier versions that old AIs have done.

[asasidh]

and that is bad because ?

[program_whiz]

the nature of the test was to see if the models can effectively compose an image of a novel concept outside the training set. If they are trained on it, it ceases to be an interesting test to some extent.

[wasabi991011]

I would urge you to re-read the blog post you are commenting on. It pretty clearly explains how it is an interesting test independently of "see[ing] if the models can effectively compose an image of a novel concept outside the training set".

[cyanydeez]

it's still interesting because there's no pelican-on-bike model, and if you're training a model well enough, then it should be obvious when a model has reached "AGI" or whatever.

[barrenko]

Would it? Tongue in cheek.

[segmondy]

It's incredible you can't reason to see if pelican on a bike is a thing. It's not! This has been discussed to death. You can ask any model to generate anything. Generate an SVG of earthworm and a robin boxing. Guess what? The smarter the model the better the image, doesn't matter if it's a vision model or not. I rolled my eyes at this eval when I first saw it, then I tried various ridiculous things and noticed a very strong correlation. Things that are absolutely not in the training set.

[podgietaru]

More to it, the actual bloody companies are using them as a reference. Maybe it’s a 3d version, not an svg - but it clearly shows they’re on the radar of these companies.

[jhalloran]

This reminded me about the news cycle last year that we were running out of training data (and how silly that was)

[HardCodedBias]

A person from Google famously put on her linkedin that her job was to optimize SVG for Gemini 3.0.

[Chu4eeno]

SVG output is useful, though. I often ask whatever LLM I have open to generate placeholder icons whenever I need them.

[OtomotO]

It's incredible people still discuss the pelicans... But then again, the ad just works.

[port3000]

Yeah I asked Nano Banana to make a render of our company office and was scarily accurate

[semilin]

They can be in the training set but not deliberately trained for. There may be a lot of people posting pelican svgs, but not typically because they're high quality and worth replicating.

[andy_xor_andrew]

Did you read the post? It's not even that long. He explicitly mentions this...

[Barbing]

Are they responding to: “I’m still not convinced that labs are training for the benchmark—if they were, I’d expect much better results.”

[Certhas]

In my reading, "training for the benchmark" is very, very different from "this benchmark is in the training data".

[drcongo]

Clearly not. There's a subset of HN users who rush to post this same thing every single time.

[Topfi]

Maybe it gets posted every time because besides a personal believe by the person popularising this "benchmark", there is no reason to assume that certain labs aren't intentionally training to game this and every other lab at least unintentionally gets improvements for this specific combination of animal and action because the internet is full of both good and bad examples, often ranked, which does inevitably become training data.

I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts. Just operating on a feeling that labs don't optimise for this (as mentioned, even if they don't training data is filled with these) is not solid enough that criticism shouldn't be leveraged when it comes up.

[simonw]

> I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts

Please share those again!

One of the things I'm most looking forward to is a lab producing a model that creates a really great pelican riding a bicycle and then a terrible sloth riding a skateboard (or whatever).

I've not seen that myself yet.

[Chu4eeno]

Evidence in the other direction (that they're able to generalize) is that I can't think of any LLM currently that can't create usable (placeholder) SVG icons, I tried a bit before the pelican became popular and it was abysmal.

[Topfi]

Happy to, here one example where Grok 4 Fast, despite producing a fairly consistent pelican [0], did severely worse in a similarly outlandish scenario along with Haiku 4.5 and GPT-5 for context: https://news.ycombinator.com/item?id=45599403

> [...] a really great pelican riding a bicycle and then a terrible sloth riding a skateboard [...]

Happy to play ball. You made a blog post a few weeks back on one of the Qwen models with the eye-catching title "Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7" [1].

Here is what Qwen3.6-35B-A3B via Openrouter provided for a sloth riding a skateboard: https://imgur.com/a/Dy8fvR5

Like Grok 4 Fasts attempt at a mushroom in a rowboat, it is barely recognisable as anything despite both Qwen3.6-35B-A3B and Grok 4 Fast having no issue with more popular (i.e. benchmarked) examples. Whether this is a case of training data being unsanitized or intentional benchmark targeted training, I cannot say, but it is the case.

And here is Opus 4.7, again via Openrouter: https://imgur.com/a/Qus1Enf

A massive delta in favour of Opus 4.7, despite the pelican Qwen3.6-35B-A3B produced being noticeably better as you rightly pointed out. What does that tell us? Whether intentional or not (with such deltas, I do have my suspicions), any eval with such a delta is clearly polluted and can not be a source of information, especially as its continued existence does hinge on you testing similar prompts in private as a sanity check, yet by your own admission never noticing the plainly apparent delta in quality. I specifically stuck with the skateboarding sloth too, to keep it as fair as possible and found this in less than 5 minutes...

I would not critique your use of this fun benchmark the way I tend to if I did not have evidence to back up my position, including private evals beyond SVGs that I can reliably use to point out major deviations between what a models claimed performance is according to major benchmarks vs the actual performance outside these known test cases.

I will also say that while I have a lot to be critical of regarding Anthropics modus operandi, especially how they present interesting findings like their j-space work, which I found was irresponsibly anthropomorphic in their reporting, especially as this wasn't a first in model interpretability, but mainly a leap due to being applied to a larger model, but of all the labs, they are the ones that never underperform my evals vs public ones and they appear to strictly keep their training data sanitised.

Happy to discuss public vs private evals and the merit of each if you'd like, I do appreciate your reporting in general but just think the SVG benches have become evidently polluted, which is also why even simple queries in my benchmarks are private. Just saw Thinking Machines Inkling model succeed in certain queries that neither Fable 5, nor GPT-5.6 Sol on any reasoning level managed, which I feel is valuable to truly gauge where we are at. Informs my work with models, my views of the industry and my assessment of the future these tools have, along with how to best implement them to enable better UX.

[0] https://simonwillison.net/2025/Sep/20/grok-4-fast/

[1] https://simonwillison.net/2026/Apr/16/qwen-beats-opus/

[Topfi]

Respectfully, did you? The comment was specific to doubting the believe simonw has that labs are not training [0] specifically for this task, which is exactly what simonw wrote in the post [1], that it is a believe of his that they don't. He did not mention any kind of evidence or any piece of information that would indicate that the commenter didn't read the blog post.

Did you read either the post or the comment it was referencing?

On the note of training on SVGs, I have seen some labs models outperform when prompted for SVGs of certain animal and action combinations (pelican on bike, panda eating burger, etc.) compared to other similarly outlandish prompts for SVG output that are not part of widely reported benchmarks, even shared evidence one of the last times this came up on here.

[0] ... incredible Simon still believes ...

[1] I’m still not convinced that labs ....

[simonw]

I'll note there's a difference between "pelicans on bikes aren't part of the training set" and "I’m still not convinced that labs are training for the benchmark".

I'm sure all sorts of crap pelican riding bicycle SVGs have ended up in the huge crawls of data that the labs feed into their pre-training steps.

What I'm questioning here is that there are labs who have sat down and deliberately tested and tweaked the performance for this particular task, independent of general model improvements.

The one exception here is Gemini, who have clearly invested a lot of effort in SVG tasks. I have no idea if my stupid benchmark influenced that decision!

Gemini have boasted about how good they are at pelicans riding bicycles, frogs on penny-farthings, giraffes driving a tiny car, ostriches on roller skates, turtles kickflipping skateboards, and dachshunds driving a stretch limousine. So if they trained for the test they did at least expand it a whole bunch! https://twitter.com/JeffDean/status/2024525132266688757

[foobarqux]

Yes that's the obvious thing to do and why straightforward variants of known tests would also be treated as contaminated by anyone being even somewhat rigorous.

I don't know why the standard is is to be sure that it is happening versus it being a plausible risk of making the results useless.

[simonw]

The pelican test has never pretended to be "rigorous". It's always openly been very much not that.

[Topfi]

> What I'm questioning here is that there are labs who have sat down and deliberately tested and tweaked the performance for this particular task, independent of general model improvements.

Given the massive delta easily reproducible with some models, is it really doubtful that certain labs have not: https://news.ycombinator.com/item?id=48951229

We are going from pretty good pelican to jumbled mess with a similarly silly, but different prompt across multiple models from multiple labs, both Western and Eastern, both Open Weight and Closed.

[oceanplexian]

Imagine if we applied this train of logic to humans.

"That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"

[program_whiz]

This is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.

[alexjplant]

We do. People who, for example, memorize question banks to pass certification tests without knowing the underlying material are equally frowned upon for not having the problem solving skills that they purport to. I'll leave the contrasts between LLMs and people to the well-written sibling comments.

[xigoi]

More like “This artist won the drawing competition because someone told her the theme in advance and the specifically practiced drawing pelicans for hundreds of hours.”

[computably]

Except, of course, LLMs are not humans, and they do not learn or "reason" in a way which even remotely resembles humans.

Plus obviously humans can still overfit to a specific style of test.