[ag315]

In most cases there are vastly more synapses than there are neurons, and beyond that the neurons and synapses are not highly rudimentary pieces but are themselves extremely complex.

It's certainly true that nervous systems do quite a bit more than language processing, but AGI would presumably also have to do quite a bit more than just language processing if we want it to be truly general.

[theptip]

I agree with the general point "we are many generations away from AGI". However, I do want to point out that (bringing this thread back to the original context) there is substantial harm that could occur from sub-AGI systems.

In the safety literature one frame that is relevant is "Agents vs. Tools/Oracles". The latter can still do harm, despite being much less complex. Tools/Oracles are unlikely to go Skynet and take over the world, but they could still plausibly do damage.

I'm seeing a common thread here of "ChatGPT doesn't have Agency (intention, mind, understanding, whatever) therefore it is far from AGI therefore it can't do real harm", which I think is a non-sequitur. We're quite surprised by how much language, code, logic a relatively simple Oracle LLM is capable of; it seems prudent to me to widen our confidence intervals on estimates of how much harm they might be capable of, too, if given the capability of interacting directly with the outside world rather than simply emitting text. Specifically, to be clear, when we connect a LLM to `eval()` on a network-attached machine (which seems to be vaguely what OpenAssistant is working towards).

[ag315]

I agree with you that it could be dangerous, but I neither said nor implied at any point that I disagree with that--I don't think the original comment was implying that either. LLM could absolutely be dangerous depending on the capabilities that we give it, but I think that's separate from questions of intentionality or whether or not it is actually AGI as we normally think of it.

[theptip]

I see, the initial reply to my G(G...)P comment, which you said was spot on, was:

> That would only be possible if Sydney were actually intelligent or possessing of will of some sort.

Which I read as claiming that harm is not possible if there is no actual intelligence or intention.

Perhaps this is all just parsing on my casual choice of words "if it was able to make outbound connections it very well might try.", in which case I'm frustrated by the pedantically-literal interpretation, and, suitably admonished, will try to be more precise in future.

For what it's worth, I think whether a LLM can or cannot "try" is about the least interesting question posed by the OP, though not devoid of philosophical significance. I like Dijkstra's quote: "The question of whether machines can think is about as relevant as the question of whether submarines can swim."

Whether or not these systems are "intelligent", what effects are they capable of causing, out there in the world? Right now, not a lot. Very soon, more than we expect.

[int_19h]

That may be so, but if that is how we define AGI, then does it really need to be one to "have anything like a mind or intentionality"?

[ag315]

I don't believe AGI needs to have actual consciousness in order to functionally be AGI, and I personally am not of the view that we will ever make a conscious computer. That said, intentionality could certainly impact the way it operates, so it's something I think is worth keeping in mind for trying to predict its behavior.