[imiric]

> So this is proof of the models actually getting stronger (previous generations of LLMs were unable to solve this one).

No, it's not.

While I don't dispute that new models may perform better at certain tasks, the fact that someone was able to use them to solve a novel problem is not proof of this.

LLM output is nondeterministic. Given the same prompt, the same LLM will generate different output, especially when it involves a large number of output tokens, as in this case. One of those attempts might produce a correct output, but this is not certain, and is difficult if not impossible for a human not expert in the domain to determine this, as shown in this thread.

[famouswaffles]

This is one of a number of such results achieved only in the last few months with only the last crop of models. They have undoubtedly gotten better in this domain. Saying anything else is just denial. You can run these same problems on GPT-4 or 5 all you want, you'll get nowhere. In fact people did, and you're hearing about it now because it's these crop of models that are getting meaningful results.

[notahacker]

As others have pointed out, a key part of the prompt used here may have been "don't search the internet" as it would most likely have defaulted to starting off with existing approaches to that problem...