[blincoln]

Nature managed to bootstrap human intelligence even though nature is not even a conscious entity, let alone a conscious entity that can describe loss functions and datasets. That seems to me like pretty solid evidence that those things are not hard requirements for creating human-level intelligence.

It sounds like maybe you're arguing that humans will never be able to conclusively determine if an artificial intelligence is equivalent to a human intelligence, on the basis of a theory that a human can't describe precisely what it is to have human-equivalent intelligence from inside the system of a human brain. Humans build things that are too complex for any one person to hold the entirety of in their conscious mind all the time, by working together, or by organizing it into simpler pieces. But even setting that tangent aside, if your theory is accurate, would you accept that an AGI could prove it was more intelligent than a human by successfully describing human intelligence and how to create an equivalent AGI?

[dartos]

Maybe? But the way AI is made nowadays is by using a human made metric to determine the quality and/or correctness of a model’s output.

Because we’re unable to make a perfect, totally correct metric, I find it unlikely that any of the current generation of AI will get anywhere near human level.

Again, not that these new models aren’t extremely useful or impressive, but not really “intelligent” as a human is.