[menssen]

I appreciate this paper for relatively clearly stating what "human-like" might entail, which in this case involves "reasoning about the causes behind other people's behavior" which is "critical to navigate the social world" as outlined in this citation:

https://www.sciencedirect.com/science/article/abs/pii/S00100...

I get frustrated often when people argue "well, it isn't really intelligent" and then give examples that are clearly dependent on our brain's chemical state and our bodies' existence in-the-physical-world.

I get the feeling that when/if we are all enslaved by a super-intelligent AI that we do not understand its motives, we will still argue that it is not intelligent because it doesn't get hungry and it can't prove to us that it has Qualia.

This paper argues that gpts are bad at understanding human risk/reward functions, which seems like a much more explicit way to talk about this, and also casts it in a way that could help reframe the debate about how human evolution and our physical beings might be significantly responsible for the structure of our rational minds.

[swatcoder]

The underlying problem is that "intelligence" is itself a crappy, poorly defined word with a fraught and inconsistent history.

It doesn't appear until the early 20th century, in the shadow of compulsory education and the challenges it presented, first as a technical label for attempts to sort students -- and later soldiers -- into the tracks in which they're most likely to succeed, and then being haphazardly asserted (but not scientifically evidenced) as some general measure of mental aptitude.

At that point it shifts from something qualitative (which mental tasks might someone be good at) to something quantitative (how much more might one personal excel at all mental tasks than another), and the burgeoning field of modern American psychology goes "Aha! A quantitative measure! Here's our meal ticket to being recognized as a science instead of those quacks from Vienna", with far too much at stake to either question the many assumptions at play or the inconsistent history of usage.

Momentum takes hold and the public takes the word into its everyday vernacular, even while it's still not a clear and sound concept in its technical domain. [Most of this is history is more academically covered in Danziger's 1987 "Naming the Mind" which is excellent, and critical foundational reading to contextualize recent hot discussions in AI]

The way you're using it when you worry about "super-intelligence" is in the sense of intelligence being some universal, unbounded, quantitative independent variable along the lines of "the more intelligent something is, the more cunningly it can pursue some rationalized goal" -- some master strategist.

That's fine, and you're not alone in that, but there's not really any sound scientific groundwork to establish that there exists some quality of the world that scales like that. You're fear, and what you try to distinguish conceptually from what the paper addresses, is an inductive leap made from highly unstable ground. It's in the same invented, purely abstract idea-space of "omnipotence" or "omniscience" where one takes a practical idea like "power to influence" or "ability to know fact" and inductively draws a line from these practical senses towards some abstract infinite/incomprehensible version of that thing. But that inductive leap a Platonic logician's parlor trick and ends up raising all kinds of abstract paradoxes, as well countless physical impracticalities about how such things could exist.

So a lot of people (academic and lay) just aren't with you in taking that framing of intelligence very seriously. For many, an "super-intelligent" software whose "motives" we don't understand is just a program that produces incorrect outputs and ought to be debugged or retired, and the more interesting questions around machine "intelligence" are practical ones like "what tasks are these programs well-suited for". Here, the authors point out that the current batch of programs are not good at tasks that benefit from a theory of mind.

Knowing the answer to that kind of question reaches back to the earliest and least disputable sense of the word, where we saw that some new students and soldiers excelled at certain tasks and struggled with others, and wanted to understand how best to educated/assign them. And likewise, as we look at these tools, the pressing question for engineers and businesses is "what are they good for and what are they not good for" rather than the fantastical "what if we make a broken program and it wants to kill everyone and we don't notice and forget to shut it off"

[thewakalix]

> and it wants to kill everyone

It wouldn't have to want to kill everyone. As long as it doesn't want to not kill everyone, the side effects of it getting what it wants could be catastrophic.

> and we don't notice

How well do we understand what's going on inside ChatGPT? How well will we understand the next?

> and forget to shut it off

Earlier I would have argued that sufficiently advanced AI could prevent itself from being shut off via Things You Didn't Expect, and would instrumentally want to preserve its existence. But these days, people are giving ChatGPT not just internet access but even actively handing it control over various processes. At this rate, the first superhuman AI will face not an impermeable box but a million conveniently labeled levers!

[TeMPOraL]

> Earlier I would have argued that sufficiently advanced AI could prevent itself from being shut off via Things You Didn't Expect

There's a good argument along these lines that I keep reposting when someone asks if we can't just shut the AI off. "All you gotta do is push a button, sir?"

https://www.youtube.com/watch?v=ld-AKg9-xpM&t=30s

[kens]

> It [the word "intelligence"] doesn't appear until the early 20th century

I'm not sure what you mean here, since the word dates back to the late 14th century with roughly the same meaning as now. Perhaps you're thinking of "intelligence quotient"?

https://www.etymonline.com/word/intelligence

[swatcoder]

Summary etymology can provide interesting reference points when looking a the history of ideas, but isn't sufficient because adjacent concepts change their meaning over time as well. It's good for showing when a word was attested and where to start looking for an understanding of how it was used and considered.

Where you say "roughly the same meaning as now" you seem to mean that "the highest faculty of the mind, capacity for comprehending general truths;" is how we think of intelligence now, but the meanings of "mind", "truth" "comprehending" and "faculties of mind" have all had their own radical shifts over the last 600 years. That quoted phrase conveys an entirely different perspective and set of assumptions/implications in the context of its time, and is not at all analogous to how we read it today.

Raymond Williams' "Keywords" collects a very interesting and accessible collection of examples of this phenomenon, although it focuses more on the language of politics and society more than the language of psychology.

The modern use of intelligence, and the conceptual constellation it represents, is essentially isolated from what's described in that article, but it's re-introduction in modern psychology does borrow from its prior existence in the lexicon.

[joe_the_user]

The way you're using it when you worry about "super-intelligence" is in the sense of intelligence being some universal, unbounded, quantitative independent variable along the lines of "the more intelligent something is, the more cunningly it can pursue some rationalized goal" -- some master strategist.

I appreciate the highlighting of the term intelligence being ill-defined. Moreover, it's certainly true that "AI safety analysts" takes intelligence as a sort magic wand term and this seems to drive their arguments.

All that said, since both computers and human brains are material artifacts, it doesn't seem impossible to create a device that combines their properties. It seems plausible that such a thing could have a variety of dangers.

For many, an "super-intelligent" software whose "motives" we don't understand is just a program that produces incorrect outputs and ought to be debugged or retired, and the more interesting questions around machine "intelligence" are practical ones like "what tasks are these programs well-suited for".

We saw early Bing Chat behave, not in ways we couldn't understand but like a deranged and vengeful human. Certainly, it was merely simulating human behavior but if today's methods produce artifacts that unselectively amplify human behaviors, it's not hard to imagine problems appearing.

We can hope that there's a fundamental difference between programs that simulate human language and programs able to plan and carry out long term goals (and carrying out long term goals is something people do so there's no good reason some kind of program couldn't do that).

I think you're right that particular weirdness of the "doomers" makes some other portion of the population dismiss concerns. But that isn't an argument that the doom isn't possible - it should be an argument to clarify how we talk of computation and human capacities (see, I don't to say "intelligence" unless I want to).

[famouswaffles]

There does seem to be a general factor of intelligence in humans though that is the single biggest indicator of performance. Yes there are other factors too.

>Here, the authors point out that the current batch of programs are not good at tasks that benefit from a theory of mind.

Not good at tasks that benefit from a theory of mind extracted from visual data.

[throwanem]

"Seem" is doing a lot of work here. So is the implicit claim that theory of mind in general can be demonstrated by current-gen foundation models, and only those aspects dependent on vision cannot.

[famouswaffles]

I say seem but it's stronger than that. all evidence and testing points towards a general factor of intelligence. The better you perform at one "kind" of intelligence task, the better you will perform at them all. The shift in defining intelligence didn't come from nowhere. Yes, It's easy to think that there are multiple different mutually exclusive-ish kinds of intelligences and that you can excel in one and it has no bearing on performance on the other but that's not really true. all indication point otherwise. I'm not saying there aren't other factors but generally, that's what you can expect.

Yes theory of mind can be demonstrated. Make up whatever bespoke story you can with characters having varying levels if intention and knowledge. Then query GPT-4 about the state of the characters.

What i'm saying is that the vision component introduces another point of error, is it a lack of theory of mind ? or being yet unable to extract the necessary features from visual data ? They rapidly learn to but Blind people who could recognize squares by feel do not have the ability to recognize squares by sight upon gaining vision. https://www.projectprakash.org/_files/ugd/2af8ef_5a0c6250cc3...

[pegasus]

"Seem" only seems to be doing lots of work. Here's a relevant article: https://en.wikipedia.org/wiki/G_factor_(psychometrics)

[menssen]

For what it's worth, I don't take that framing of intelligence seriously either. It's useful to have a word to describe the far-future state of the increasing capabilities of Computers.

I'm just saying that I don't think there's any point on that line where we will be comfortable admitting that the machine is "intelligent" or "conscious" or "AGI," or whatever, and that I appreciate attempts to quantify (or at least qualify) what we MEAN when we say that, rather than just goalpost-moving.

[pixl97]

Most of what you're saying here is describing the alignment issue.

We (mostly) don't want unaligned A(G|S)I. The outcomes of that could be extenstential.

[swatcoder]

Only for those mundane senses of alignment where we say "This system is reliable in tasks that look like X and unreliable in tasks that look like Y, so let's craft hard boundaries to avoid naive use for Y"

But it's skeptical of the other sense alignment, where a potential Master Strategist needs to be trained or crippled before it outsmarts us. It sees that perspective as comparable to logicians debating whether we might live in the domain of a benevolent or evil omnipotence: "if an ant is more powerful than a rock, and I'm more powerful than an ant, then perhaps there is something so powerful that it encompasses all opportunities to influence the universe including the power to hide itself from me." -- which comes from taking a concrete measure, assuming that it's an independent variable, and then inductively extending it to an infinite or otherwise unevidenced scale. This technique is undisprovable and so it's easy for "rational" people to mine work from it for a very long time, but history and analysis give room for skeptics to be like "WTF you going on about; let's have some tea"

[pixl97]

" but history and analysis give room for skeptics to be like

When the skeptic is correct. The problem with skeptics is when incorrectness is not terminal, they can't hear you over the sound of pushing the goal posts farther to give a reasonable rebuttal for their originally incorrect statements.

[mistermann]

> I get frustrated often when people argue "well, it isn't really intelligent" and then give examples that are clearly dependent on our brain's chemical state and our bodies' existence in-the-physical-world.

A big part of the problem is that "is" has a wide variety of inconsistent meanings, and that this fact is sub-perceptual, and that it is culturally very inappropriate to comment on aspects of our culture like this, preventing knowledge of the problem from spreading.

/u/Swatcoder makes essentially the same point but in much more detail, though regarding less important words.