[vessenes]

Thanks for this - Looking forward to reading the full paper.

That said, the most obvious objection that comes to mind about the title is that … well, I feel that I’m generally intelligent, and therefore general intelligence of some sort is clearly not impossible.

Can you give a short précis as to how you are distinguishing humans and the “A” in artificial?

[catoc]

That about ‘cogito ergo sums it up’ doesn’t it?

Intelligence is clearly possible. My gut feeling is our brain solves this by removing complexity. It certainly does so, continuously filtering out (ignoring) large parts of input, and generously interpolating over gaps (making stuff up). Whether this evolved to overcome this theorem I am not intelligent enough to conclude.

[giardini]

catoc states, amongst other things, that: >"Intelligence is clearly possible."<

Perhaps not a citation but a proof is required here!

[catoc]

Clearly possible in humans - the statement in the parent I was replying to.

I would indeed definitely like to see proof - mathematical or applied - of in silico intelligence

[eru]

Do you expect silicon to be less capable of the necessary computation than cells?

[giardini]

At best, I'm asking for a clear scientific definition of intelligence. At worst I'm questioning its very existence.

[filoeleven]

If you haven't yet encountered it, check out Michael Levin's lab and work. Among other things, they are trying to figure out what "basal cognition" is; the idea being that even if we can't point to some dividing line in the end, we'll have a better understanding of what cognition is and where it shows up. And it shows up surprisingly far down!

The definition of "intelligence" that he works with comes from William James: the ability to achieve the same goal by different means. It's a useful definition, given the remarkable stuff coming out of his lab.

[giardini]

I would indeed definitely like to see proof - of intelligence!

[ICBTheory]

Sure I can (and thanks for writing)

Well, given the specific way you asked that question I confirm your self assertion - and am quite certain that your level of Artificiality converges to zero, which would make you a GI without A...

- You stated to "feel" generally intelligent (A's don't feel and don't have an "I" that can feel) - Your nuanced, subtly ironic and self referential way of formulating clearly suggests that you are not a purely algorithmic entity

A "précis" as you wished: Artificial — in the sense used here (apart from the usual "planfully built/programmed system" etc.) — algorithmic, formal, symbol-bound.

Humans as "cognitive system" have some similar traits of course - but obviously, there seems to be more than that.

[kevin42]

>but obviously, there seems to be more than that.

I don't see how that's obvious. I'm not trying to be argumentative here, but it seems like these arguments always come down to a qualia, or the insistence that humans have some sort of 'spark' that machines don't have, therefore: AGI is not possible since machines don't have it.

I also don't understand the argument that "Your nuanced, subtly ironic and self referential way of formulating clearly suggests that you are not a purely algorithmic entity". How does that follow?

What scientific evidence is there that we are anything other than a biochemical machine? And if we are a biochemical machine, how is that inherently capable of more than a silicon based machine is capable of?

[ben_w]

> I also don't understand the argument that "Your nuanced, subtly ironic and self referential way of formulating clearly suggests that you are not a purely algorithmic entity". How does that follow?

It doesn't follow.

Trivially demonstrated by the early LLM that got Blake Lemonie to break his NDA also emitting words which suggested to Lemonie that the LLM had an inner life.

Or, indeed, the output device y'all are using to read/listening to my words, which is also successfully emitting these words despite the output device very much only following an algorithm that simply recreates what it was told to recreate. "Ceci n'est pas une pipe", etc. https://en.wikipedia.org/wiki/The_Treachery_of_Images

[ICBTheory]

Oh no, I am not at all trying to find an explanation of why this is (qualia etc.). There is simply no necessity for that. It is interesting, but not part of the scientific problem that i tried to find an answer to.

The proof (all three of them) holds without any explanatory effort concerning causalities around human frame-jumping etc.

For this paper, It is absolutely sufficient to prove that a) this cannot be reached algorithmically and that b) evidence clearly shows that humans can (somehow) do this , as they have already done this (quite often).

[fc417fc802]

> this cannot be reached algorithmically

> humans can (somehow) do this

Is this not contradictory?

Alternatively, in order to not be contradictory doesn't it require the assumption that humans are not "algorithmic"? But does that not then presuppose (as the above commenter brought up) that we are not a biochemical machine? Is a machine not inherently algorithmic in nature?

Or at minimum presupposes that humans are more than just a biochemical machine. But then the question comes up again, where is the scientific evidence for this? In my view it's perfectly acceptable if the answer is something to the effect of "we don't currently have evidence for that, but this hints that we ought to look for it".

All that said, does "algorithmically" here perhaps exclude heuristics? Many times something can be shown to be unsolvable in the absolute sense yet readily solvable with extremely high success rate in practice using some heuristic.

[stevenhuang]

OP seems to have a very confused idea of what an algorithmic process means... they think the process of humans determining what is truthful "cannot possibly be something algorithmic".

Which is certainly an opinion.

> whatever it is: it cannot possibly be something algorithmic

https://news.ycombinator.com/item?id=44349299

Maybe OP should have looked at a dictionary for what certain words actually mean before defining them to be something nonsensical.

[eru]

> Maybe OP should have looked at a dictionary for what certain words actually mean before defining them to be something nonsensical.

Making non-standard definitions of words isn't necessarily bad, and can be useful in certain texts. But if you do so, you need to make these definitions front-and-centre instead of just casually assuming your readers will share your non-standard meaning.

And where possible, I would still use the standard meanings and use newly made up terms to carry new concepts.

[okokok0k]

Maybe you need to update an outdated model?

Nothing in physics requires us to use your prior experience as some special epoch.

Meaning is mutable social relationship as language meaning is not immutable physics.

[dragonwriter]

> Alternatively, in order to not be contradictory doesn't it require the assumption that humans are not "algorithmic"? But does that not then presuppose (as the above commenter brought up) that we are not a biochemical machine? Is a machine not inherently algorithmic in nature?

No, computation is algorithmic, real machines are not necessarily (of course, AGI still can't be ruled out even if algorithmic intelligence is, only AGI that does not incorporate some component with noncomputable behavior.)

[JumpCrisscross]

> computation is algorithmic, real machines are not necessarily

Author seems to assume the latter condition is definitive, i.e. that real machines are not, and then derive extrapolations from that unproven assumption.

[fc417fc802]

> No, computation is algorithmic, real machines are not necessarily

As the adjacent comment touches on are the laws of physics (as understood to date) not possible to simulate? Can't all possible machines be simulated at least in theory? I'm guessing my knowledge of the term "algorithmic" is lacking here.

[kolinko]

Using computation/algorithmic methods we can simulate nonalgorithmic systems. So the world within a computer program can behave in a nonalgorithmic way.

Also, one might argue that universe/laws of physics are computational.

[Delk]

> For this paper, It is absolutely sufficient to prove that a) this cannot be reached algorithmically and that b) evidence clearly shows that humans can (somehow) do this , as they have already done this (quite often).

The problem with these kinds of arguments is always that they conflate two possibly related but non-equivalent kinds of computational problem solving.

In computability theory, an uncomputability result essentially only proves that it's impossible to have an algorithm that will in all cases produce the correct result to a given problem. Such an impossibility result is valuable as a purely mathematical result, but also because what computer science generally wants is a provably correct algorithm: one that will, when performed exactly, always produce the correct answer.

However, similarly to any mathematical proof, a single counter-example is enough to invalidate a proof of correctness. Showing that an algorithm fails in a single corner case makes the algorithm not correct in a classical algorithmic sense. Similarly, for a computational problem, showing that any purported algorithm will inevitably fail even in a single case is enough to prove the problem uncomputable -- again, in the classical computability theory sense.

If you cannot have an exact algorithm, for either theoretical or practical reasons, and you still want a computational method for solving the problem in practice, you then turn to heuristics or something else that doesn't guarantee correctness but which might produce workable results often enough to be useful.

Even though something like the halting problem is uncomputable in the classical, always-inevitably-produces-correct-answer-in-finite-time sense, that does not necessarily stop it from being solved in a subset of cases, or to be solved often enough by some kind of a heuristic or non-exact algorithm to be useful.

When you say that something cannot be reached algorithmically, you're saying it's impossible to have an algorithm that would inevitably, systematically, always reach that solution in finite time. And you would in many cases be correct. Symbolic AI research ran into this problem due to the uncomputability of reasoning in predicate logic. (Uncomputability is not the main problem that symbolic AI ran into but it was one of them.)

The problem is that when you say that humans can somehow do this computationally impossible thing, you're not holding human cognition or problem solving to the same standard of computational correctness. We do find solutions to problems, answers to questions, and logical chains of reasoning, but we aren't guaranteed to.

You do seem to be aware of this, of course.

But you then run into the inevitable question of what you mean by AGI. If you hold AGI to the standard of classical computational correctness, to which you don't hold humans, you're correct that it's impossible. But you have also proven nothing new.

A more typical understanding of AGI would be something similar to human cognition -- not having formal guarantees but working well enough for operating in, understanding, or producing useful results the real world. (Human brains do that well in the real world -- thanks to having evolved in it!)

In the latter case, uncomputability results do not prove that kind of AGI to be impossible.

[eru]

Indeed. And it's fairly trivial to see that computability isn't the right lens to view intelligence through:

The classic Turing test takes place over a finite amount of time. Normally less than an hour, but we can arbitrarily give the interlocutor, say, up to a week. If you don't like the Turing test, then just about any other test interaction we can make the system undergo will conclude below some fixed finite time. After all, humans are generally intelligent, even if they only get a handful of decades to prove it.

During that finite time interaction, only a finite amount of interaction will be exchanged.

Now in principle a system could have a big old lookup table with all prefixes of all possible interactions as keys, and values are probability distributions for what to send back next (and how long to wait before sending the reply). That table would be finite. And thus following it would be computable.

Of course, the table would be more than astronomical in size, and utterly impossible to manifest in our physical universe. But computability is too blunt an instrument to formalise this with.

In the real universe, you would need to _compress_ that table somehow, eg in a human brain or perhaps in an LLM or so. And then you need to be able to efficiently uncompress the parts of the table you need to produce the replies. Whether that's possible and how are all questions of complexity theory, not computability.

See Scott Aaronson's excellent 'Why Philosophers Should Care About Computational Complexity': https://arxiv.org/abs/1108.1791

[somenameforme]

Consciousness is an issue. If you write a program to add 2+2, you probably do not believe some entity poofs into existence, perceives itself as independently adding 2+2, and then poofs out of existence. Yet somehow, the idea of an emergent consciousness is that if you instead get it to do 100 basic operations, or perhaps 2^100 then suddenly this becomes true? The reason one might believe this is not because it's logical or reasonable - or even supported in any way, but because people assume their own conclusion. In particular if one takes a physicalist view of the universe then consciousness must be a physical process and so it simply must emerge at some sufficient degree of complexity.

But if you don't simply assume physicalism then this logic falls flat. And the more we discover about the universe, the weirder things become. How insane would you sound not that long ago to suggest that time itself would move at different rates for different people at the same "time", just to maintain a perceived constancy of the speed of light? It's nonsense, but it's real. So I'm quite reluctant to assume my own conclusion on anything with regards to the nature of the universe. Even relatively 'simple' things like quantum entanglement are already posing very difficult issues for a physicalist view of the universe.

[pixl97]

>Yet somehow, the idea of an emergent consciousness is that if you instead get it to do 100 basic operations, or perhaps 2^100 then suddenly this becomes true

Why not? You can do a simple add with assembly language in a few operations. But if you put millions and millions of operations together you can get a video game with emergent behaviors. If you're just looking at the additions, where does the game come from? Is it still a game if it's not output to a monitor but an internal screen buffer?

[somenameforme]

You're not speaking of a behavior but of a "thing." Your consciousness sits idly inside your body, feeling as thought it's the driving all actions of its free will. There's no necessity, reason, or logical explanation for this thing to exist, let alone why or where it comes from.

No matter how many instructions you might use to create the most compelling simulation of a dragon in a video game, neither that dragon or any part of it is going to poof into existence. I'm sure this is something everybody would agree with. Yet with consciousness you want to claim 'well except its consciousness, yeah that'll poof into existence.' The assumption of physicalism ends up requiring people to make statements that they themselves would certainly call absurd if not for the fact that they are forced to make such statements because of said assumption!

And what is the justification for said assumption? There is none! As mentioned already quantum entanglement is posing major issues for physicalism, and I suspect we're really only just beginning to delve into the bizarro nature of our universe. So people embrace physicalism purely on faith.

[pixl97]

>There's no necessity, reason, or logical explanation for this thing

I mean, I disagree. It's a internal virtual 'playground' you can bounce ideas off of and reason against. Obviously it imparts some survival benefits to creatures that have one at this point in evolution.

[kevin42]

My issue is that from a scientific point of view, physicalism is all we have. Everything else is belief, or some form of faith.

Your example about relativity is good. It might have sounded insane at some point, but it turns out, it is physics, which nicely falls into the physicalism concept.

If there is a falsifiable scientific theory that there is something other than a physical mechanism behind consciousness and intelligence, I haven't seen it.

[somenameforme]

I don't think science and consciousness go together quite well at this point. I'll claim consciousness doesn't exist. Try to prove me wrong. Of course I know I'm wrong because I am conscious, but that's literally impossible to prove, and it may very well be that way forever. You have no way of knowing I'm conscious - you could very well be the only conscious entity in existence. This is not the case because I can strongly assure you I'm conscious as well, but a philosophical zombie would say the same thing, so that assurance means nothing.

[kevin42]

There are more than one theories, as well as some evidence that consciousness may not exist in the way we'd like to think.

It may be a trick our mind plays on us. The Global Workspace Theory addresses this, and some of the predictions this theory made have been supported by multiple experiments. If GWT is correct, it's very plausible, likely even, that an artificial intelligence could have the same type of consciousness.

[ben_w]

Boltzmann brains and A. J. Ayer's "There is a thought now".

Ages ago, it occurred to me that the only thing that seemed to exist without needing a creator, was maths. That 2+2 was always 4, and it still would be even if there were not 4 things to count.

Basically, I independently arrived at similar conclusion as Max Tegmark, only simpler and without his level of rigour: https://benwheatley.github.io/blog/2018/08/26-08.28.24.html

(From the quotation's date stamp, 2007, I had only finished university 6 months earlier, so don't expect anything good).

But as you'll see from my final paragraph, I no longer take this idea seriously, because anything that leads to most minds being free to believe untruths, is cognitively unstable by the same argument that applies to Boltzmann brains.

MUH leads to Aleph-1 infinite number of brains*. I'd need a reason for the probability distribution over minds to be zero almost everywhere in order for it to avoid the cognitively instability argument.

* if there is a bigger infinity, then more; but I have only basic knowledge of transfinites and am unclear if the "bigger" ones I've heard about are considered "real" or more along the lines of "if there was an infinite sequence of infinities, then…"

[eru]

Human minds are fairly free to believe untruths. At least to a certain extent: it's rather hard to _really_ believe things that contradict your lived experience.

You can _say_ that you believe them, but you won't behave as if you believe them.

[ben_w]

Yeah, that's fair, I phrased that part wrong.

The problem with Boltzmann brains is that, by construction, they're going to have incorrect beliefs about almost everything.

Like, imagine watching a TV tuned to a dead station and somehow the random background noise looked and sounded like someone telling you the history of the world, and it really was just random noise doing this — that level of being wrong about almost everything.

Not even just errors like believing 1+1=3, but that this is equally likely as believing incoherent statements like 1+^Ω[fox emoji].

[bluefirebrand]

> What scientific evidence is there that we are anything other than a biochemical machine? And if we are a biochemical machine, how is that inherently capable of more than a silicon based machine is capable of

Iron and copper are both metals but only one can be hardened into steel

There is no reason why we should assume a silicon machine must have the same capabilities as a carbon machine

[vidarh]

Unless you can show - even a single example would do - that we can compute a function that is outside the Turing computable set, then there is a very strong reason that we should assume a silicon machine has the same capabilities as a carbon machine to compute.

[nyrikki]

The problem is that your challenge is begging the question.

Computability or algorithms are the problem.

It is all the 'no effective algorithm exists for X' that is the problem.

Spike train retiming and issues with riddled basins in existing computers and math is an example if you drop compute a function

[ben_w]

> There is no reason why we should assume a silicon machine must have the same capabilities as a carbon machine

Then make your computer out of carbon.

While the broader principle, that we don't know what we're doing and AI as it currently exists is a bit cargo-culty, this is a critique of the SOTA and is insufficient to be generalised: we can reasonably say "we probably have not", we can't say "we definitely cannot ever".

Who knows, perhaps our brains do somehow manage to do whacky quantum stuff despite seeming to be far too warm and messy for that. But even that is just an implementation detail.

[eru]

> Who knows, perhaps our brains do somehow manage to do whacky quantum stuff despite seeming to be far too warm and messy for that. But even that is just an implementation detail.

Yes. And we are pretty close to building practical quantum computers. Though so far, we haven't really found much they would be good for. The most promising application seems to be for simulating quantum systems for material science.

[dr_dshiv]

Yeah, but bronze also makes great swords… what’s the point here?

[john-h-k]

> You stated to "feel" generally intelligent (A's don't feel and don't have an "I" that can feel) - Your nuanced, subtly ironic and self referential way of formulating clearly suggests that you are not a purely algorithmic entity

This is completely unrelated to the proof in the link. You have to clearly explain what reasoning in your argument for “AGI is impossible” also implies human intelligence is possible. You can’t just jump to conclusions “you sound human therefore intelligence is possible”

[_0ffh]

It's simple: Either your proof holds for NGI as much as for AGI, or neither, or you can clearly define what differentiates them that makes it work for one and not the other.

[stevenhuang]

These are.. very weak rebuttals.

[vessenes]

Agreed. I thought my followup qs were fair. I'd like to understand the argument, but the first response makes me think it's not worth wading too deeply in.

[madaxe_again]

I think you’ve just successfully proven that general human intelligence indeed does not exist.

[vessenes]

So, in a word: a) there is no ghost in the machine when the machine is a formal symbol-bound machine. And b) to be “G” there must be a ghost in the machine.

Is that a fair summary of your summary?

If so do you spend time on both a and b in your papers? Both are statements that seem to generate vigorous emotional debate.

[nextaccountic]

> level of Artificiality

How do you define that? And why is this important?

[rusk]

Not the person asked, but in time honoured tradition I will venture forth that the key difference is billions of years of evolution. Innumerable blooms and culls. And a system that is vertically integrated to its core and self sustaining.

[ben_w]

AI can be, and often are, trained by simulated evolution.

[rusk]

Simulated.

[ben_w]

You have to say why you think that matters. It still culls the unfit.

[rusk]

I don’t. You have boiled a process of billions of years down to a single sentence. You should ponder your absurdity.

[ben_w]

That's the point of language, to abstract a complex thing to what is often as little as a single word or sentence. It's not like the idea represented by the words "simulated evolution" is itself as simple as those two words anyway.

That it takes nature "billions of years" for natural evolution isn't even important here, because it's not like simulations have to run in real-time.

If you run simulated evolution with mechanical parts and the reward function of things that function like clocks, you get the (design of) a thing that functions like a clock, and if you run the physics simulation of the design, you can tell the time with it. Do it with electronics and things that act like a radio, you get a radio. Do it with a CAD design and the goal of strength for minimum mass, you end up with something that looks bone-like.

We also do it with AI, why should we expect it not to produce things in the general category of "minds"? Not necessarily human minds, even the biggest by parameter count are much smaller structures than our brains, but the general category.

[jemmyw]

I would argue that you are not a general intelligence. Humans have quite a specific intelligence. It might be the broadest, most general, among animal species, but it is not general. That manifests in that we each need to spend a significant amount of time training ourselves for specific areas of capability. You can't then switch instantly to another area without further training, even though all the context materials are available to you.

[Tadpole9181]

This seems like a meaningless distinction in context. When people say AGI, they clearly mean "effectively human intelligence". Not an infallible, completely deterministic, omniscient god-machine.

[jemmyw]

There's a great deal of space between effectively human and god machine. Effectively human meaning it takes 20 years to train it and then it's good at one thing and ok at some other things, if you're lucky. We expect more from LLMs right now, like being able to have very broad knowledge and be able to ingest vastly more context than a human can every time they're used. So we probably don't just think of or want a human intelligence.. or we want an instant specific one, and the process of being about to generate an instant specific one would surely be further down the line to your god like machine anyway.

[const_cast]

The measure of human intelligence is never what humans are good at, but rather the capabilities of humans to figure out stuff they haven't before. Meaning, we can create and build new pathways inside our brains to perform and optimize tasks we have not done before. Practicing, then, reinforces these pathways. In a sense we do what we wish LLMs could - we use our intelligence to train ourselves.

It's a long (ish) process, but it's this process that actually composes human intelligence. I could take a random human right now and drop them somewhere they've never been before, and they will figure it out.

For example, you may be shocked to know that the human brain has no pathways for reading, as opposed to spoken language. We have to manually make those. We are, literally, modifying our brains when we learn new skills.

[jemmyw]

> For example, you may be shocked to know that the human brain has no pathways for reading, as opposed to spoken language.

I'm not shocked at all.

> I could take a random human right now and drop them somewhere they've never been before, and they will figure it out.

Yes, well not really. You could drop them anywhere in the human world, in their body. And even then, if you dropped me into a warehouse in China I'd have no idea what to do, I'd be culturally lost and unable to understand the language. And I'd want to go home. So yes you could drop in a human but they wouldn't then just perform work like an automonon. You couldn't drop their mind into a non human body and expect anything interesting to happen, and you certainly couldn't drop them anywhere inhospitable. Nearer to your example, you couldn't drop a football player into a maths convention and a maths professor into a football game and expect good results. The point of an AI is to be useful. I think AGI is very far away and maybe not even possible, whereas specific AIs are already abound.

[andoando]

It doesn't take 20 years for humans to train new tasks. Perhaps to master very complicated tasks, but there is many tasks you can certainly learn to do in a short amount of time. For example, "Take this hammer, and put nails in top 4 corners of this box, turn it around, do the same". You can master that relatively easy. An AGI ought to be able to practically all such tasks.

In any case, general intelligence merely means the capability to do so, not the amount of time it takes. I would certainly bet a physical theorist for example can learn to code in a matter of days despite never having been introduced to a computer before, because our intelligence is based on a very interconnected world model.

[jemmyw]

It takes about 10 years to train a human to do anything useful after creation.

[andoando]

A 4 year old can navigate the world better than any AI robot can

[nextaccountic]

Note that AGI (artificial general intelligence) and ASI (artificial superintelligence) are different thins

AGI reaches human level and ASI goes beyond that