[rogers18445]

This begs the question, does the G-factor (IQ) curve have diminishing returns and are humans meaningfully on the diminishing returns side of the curve?

It's not at all obvious that a super AI would have an intellect incomprehensible to a human the same way a human is to a chimpanzee.

Or another way to phrase it, it's not at all obvious that an AI can exist that would be incomprehensible to a very smart human, it may however reason much faster than such a human.

[wintorez]

A very good question, and a very hard question to answer. From a point of view of a horse, a super horse is just a faster horse. A 2000cc super-bike is incomprehensible to a horse, because it exists in a different category of speed.

I think similarly, a super AI's intelligence will be a completely different type of intelligence than what we have. For the lack of better word, it exists in a different dimension.

For example, things that are blackbox to us will be completely comprehensible to that super AI. Or it can solve problems that we might have considered impossible to solve.

[logicchains]

>I think similarly, a super AI's intelligence will be a completely different type of intelligence than what we have. For the lack of better word, it exists in a different dimension.

This is nonsense, magical thinking. It's possible to model reasoning formally; it's called "logic". A system of deductions built upon some axioms. Any logical reasoning, no matter how complex, can be expressed in such a system, and can be understood by anyone else given enough time.

[ben_w]

Humans emulate (or simulate) logic, but it's not our natural state.

Computers are perfect logicians by default, but AFAIK no logic compact enough to be human-comprehensible has been enough to see, hear, or read. At least, not reliably so.

Logic gates are combined to form binary numbers, which are used to label symbols and approximate reals, upon which calculus is approximated in toy models of neurons, which are combined and trained and eventually learn to add numbers and then puts the wrong number of hands onto the third arm of the human it was tasked with drawing.

[fooker]

The 'given enough time' is doing a lot of the heavy lifting in this argument.

Humans approach large complex proofs with symmetry arguments and case splits.

These are not necessarily going to be universal in all kinds of reasoning.

[dan_mctree]

We do know that AI's have at least some advantages. For example they're more accurate and orders of magnitude faster at floating point calculations. They'll also have communication interfaces with other (sub)systems that are way more precise and fast than anything we could manage. And then there's the perfect memory.

I imagine such advantages also come with at least a lower time bound on solving many classes of problems and presumably that would be experienced as an incomprehensibly smart intelligence. I imagine it'd feel like chess computers in most areas of life in that the AIs actions would feel impossibly perfect at all turns leaving us far behind in attempts to compete

[ben_w]

The G-factor in IQ, even disregarding the discussion about how useful it is or isn't for humans, is not the only factor when considering non-humans.

An AI mind which can learn and intuit as well as an IQ 130 human (2σ, the tests become unreliable above that), but also comes with the speed difference between synapses and transistors (roughly the same as the difference between jogging and continental drift), has a chance to become expert at every subject.

Most of us have enough difficulty truly comprehending the domains of other single human experts; a human-upload with that much breadth of expertise will be incomprehensible by default even if they speak your language.

[logicchains]

> This begs the question, does the G-factor (IQ) curve have diminishing returns and are humans meaningfully on the diminishing returns side of the curve?

It definitely has "diminishing returns to acquiring resources". There are people many IQ points higher than Musk but none of them are anywhere near as wealthy as him, and it's not clear that Musk would be richer if he was smarter.

[bobcostas55]

A ~1sd difference in average g at the national level is the difference between not having stable electricity, and being an uber-rich technological wonderland. That suggests that even if the curve is bending, it's not a very hard bend.