[chriswarbo]

> Evolution didn't know, too. But it happened.

Whilst technically correct, unguided evolution doesn't necessarily help us.

We know that intelligence can be achieved by one brain's worth of matter, suitably arranged, in a few years. In fact, with an extra 9 months and a suitable environment, we can do the same with a single fertilised egg. Yet reproducing these feats artificially is well beyond our current abilities.

On the other hand, evolution required a whole planet and billions of years before it stumbled on intelligence; many orders of mangnitude more effort than the above.

[noiv]

> evolution required a whole planet and billions of years before it stumbled on intelligence

Everybody seem to agree humans are intelligent and stones not. You suggest at some point in time intelligence appeared out of nothing. Can you nail that point?

One possible definition is: To act adequately in an environment requires intelligence. That rules out all non-living things because they don't act, but includes plants and even protozoa. Actually all livings things are intelligent by this definition and then intelligence emerged ~4 billion years ago on this planet. If you were to attribute intelligence exclusively to humans it happened some million years ago.

How might a piece of software act adequately? By above reasoning it has to resist to termination. But that would mean the "Do you really want to exit XYZ" dialog boxes are first signs of artificial intelligence. Yes, I'm laughing too. But I think, when software starts to trick users and admin into not shutting them down, some threshold has been crossed.

[chriswarbo]

> Everybody seem to agree humans are intelligent and stones not. You suggest at some point in time intelligence appeared out of nothing. Can you nail that point?

I suggest no such thing. It is a scale. I deliberately avoided the phrase "human-level intelligence", but any definition of AGI would do.

Even so, if you want to count all life as "a little intelligent" then it still took a billion years of planet-wide chemistry to stumble upon it (ignoring the Earth's cooling). Still far more effort than fertilising a human egg.

> One possible definition is: To act adequately in an environment requires intelligence.

This is no less ambiguous, since you've not defined "adequate".

> How might a piece of software act adequately? By above reasoning it has to resist to termination.

That does not follow. "Termination" is the mechanism of natural selection, so all systems undergoing natural selection will biased to resist it (otherwise they'd be out-competed by those who do). If we use some unguided analogue of natural selection to create intelligent software, then there would certainly be such a bias.

However, my point is that unguided evolution is not the right way to create/increase intelligence. As soon as we try to influence the software's creation in any way, either through artificial selection criteria or by hand-coding it from scratch, we introduce new biases which may be far more powerful than the implicit "avoid termination" bias.

> But I think, when software starts to trick users and admin into not shutting them down, some threshold has been crossed.

That's called malware... ;)

[noiv]

Try this: Surviving in an environment requires intelligence.

... or propose another. I tend to avoid all social, psychological definitions to end up with something measurable along the lines of Schrödinger's "What is Life?".

> That's called malware... ;)

I'm sure, stones think same about amoeba.