You are looking at it from a product perspective.
From a scientific perspective, it just means the respective benchmark is meaningless, so we don't know how well such a model generalizes.
Not so! From a scientific perspective the result you can achieve matters, no one is a blank slate.
For humans this is true as well. The way you teach matters. Look at how the bell curve got absolutely demolished for example when math was taught this way:
Another way to look at this is: The first assembly language compiler was handcoded in binary to begin with, and then that compiler's machine code was translated to the more expressive language (assembly). Similar for Fortran/C/etc. from assembly code. Progressively, more expressive languages have been bootstrapped from prior lower-level languages. In a similar way, perhaps a more concise LLM can be built by utilizing a less efficient one?
For humans this is true as well. The way you teach matters. Look at how the bell curve got absolutely demolished for example when math was taught this way:
https://archive.nytimes.com/opinionator.blogs.nytimes.com/20...