Worry not, Pelicans on bicycles had been ranking pretty high on your favorite search engine for a while. I struggle to imagine a world in which it was not already scraped and turned into training data by at least one provider:
1. Models need to be good at the questions we ask them, not the questions we could ask them.
2. The questions, at least partially, are correlated with information people consume.
3. People mostly consume viral content.
4. Ergo you should scrape viral content for training data.
Probably it was added to the training data on the first day when this benchmark was on HN main page. It’s a bad benchmark since then. I don’t know why people still rate it high. Basically, every benchmark becomes pointless after it was published. They are good only to have a picture at the time they’re published first, and not after.
Imagine you never in your life seen a pelican or any other bird. You only read about them in text. Detailed descriptions but still just descriptions. And now you would need to one shot draw it. Might be a bit hard.
The model likely does know what a pelican looks like; the hard part is translating that into ordered SVG path commands. It's really testing spatial and vector composition, not biological recall.
1. Models need to be good at the questions we ask them, not the questions we could ask them.
2. The questions, at least partially, are correlated with information people consume.
3. People mostly consume viral content.
4. Ergo you should scrape viral content for training data.