[ben_w]

Half agree. Also, there's already some tests like this:

• https://www.sciencedaily.com/releases/2026/01/260125083356.h...

• https://creativitybenchmark.ai

But the "half" is that in practice, "creativity" is finding the border zone between novelty and familiarity. Every time I've been praised for it in my life I've had the inside view on what inspired me, and once I'd seen enough culture I started to be able to spot the sources of inspiration in much other work too. How Star Trek is inspired by mixing cold war naval manoeuvres, the age of exploration, and John W. Campbell's "Islands of Space", and how Islands of Space itself feels like Jules Verne with less autism (Verne had a lot of lists) and more civilising-mission smugness.

Go too far outside the border zone and you get the same initial reaction as "Danse macabre" by Saint-Saëns ("horrible screeching from solo violin" causing widespread feelings of anxiety). It took familiarity for it to be seen as it is now, "one of Saint-Saëns' masterpieces, widely regarded and reproduced in both high and popular culture" to quote Wikipedia.

[codingdave]

Well, now you are including cultural acceptance as a requirement for creativity, which is definitely not a given. As you were saying, many new works take time to gain acceptance, but that does not mean they were not creative works.

I think we're talking about two different types of "creative". One is the re-hashing of existing ideas and content into existing structures. That styles is the kind you see in marketing, design, the corporate world, and even much of the artistic world. It is also the kind in the links you shared, which specifically are measuring performance on "well-defined tasks". And there is nothing wrong with that type of creativity. But there is a second kind - the kind where people bring unique ideas to the world that do not match what has come before. We don't even need to stick to art for that, we can take a fairly obvious example of someone like Einstein, who was able to devise new theories based on existing knowledge - theories that were not re-hashes of the current ideas, but completely new ways to approach the same information.

If LLMs can do that - take the same body of information, from any subject area, and devise novel ideas that do not match pre-existing structure, building something more than the sum of the parts, that is the more interesting style of creativity to measure

[ben_w]

I don't understand your example for the second kind. Even with your own words, "Einstein, who was able to devise new theories based on existing knowledge", that sounds to me much the same as the first kind, because of e.g. GR coming from Riemann geometry, and SR coming from Lorentz transformations.

Looking back at history, it's kinda remarkable that the ancient Greeks, who knew the surface of the Earth was curved, didn't realise this curvature itself was a contradiction of Euclid's triangle angle sum. Likewise, should it be difficult to invent discrete spaces, once someone has asked if atoms exist? Evidentially both were difficult, as the gap was thousands of years.