[guygurari]

Achieve a significant scientific or mathematical breakthrough without human supervision. Domain experts should agree that the new result is truly groundbreaking, and achieving it required fundamentally new ideas — not merely interpolating existing results.

Examples of discoveries that would have counted if they weren’t already made: Relativity (say a derivation of E=mc^2), Quantum Mechanics (say a calculation of the hydrogen energy levels), the discovery of Riemmannian geometry, the discovery of DNA, and the discovery of the theory of evolution with natural selection.

The idea is to test the system’s out of distribution generalization: its ability to achieve tasks that are beyond its training distribution. This is something that humans can do, but no current LLM appears to be able to do.

[famouswaffles]

"Supercharged Interpolation" is not a thing that exists sorry.

Learning in high dimensions always results in extrapolation.

https://arxiv.org/abs/2110.09485

And I would love to see the human that can generalize beyond their training distribution.