[joe_the_user]

Once someone is fluent in a language, the logical operations and judgements involved stop being overt and highly visible to the conscious mind. But that doesn't mean that one stops getting the benefits and results of logical operations.

What you might see as logical operations "not mattering", I would see as logical operations integrated so deeply into reflexive operations that it's hard to see where one ends and the other begins. The contrast is that humans can do pattern recognition in a neural net fashion, taking something like the multidimensional average of a set of things. But a human can also receive a language-level input that some characteristic is or isn't important for recognizing a given thing and incorporate that input into their broad-average concepts. That kind of thing can't be done by deep learning currently - well, not a non-kludgey sort of way.

Similarly, I have a soft-spot for the view that a mind is only as good as its set of inputs.

It depends on how you want to mean that. A human can take inputs on one thing and apply them seamlessly to another thing. Neural nets tend to be very dependent on the task-focused content fed them.

[visarga]

I think "a better set of inputs" is the real world or much better simulators to train our RL agents. François Chollet (author of Keras) was saying a similar thing - focusing too much on architectures and algorithms we forget the importance of the environment, an agent can only become as smart as the hardest problem it has to solve in its environment, and depends on the richness of said environment for learning. Humans are not general intelligences either, we're just good at being human (surviving in our environment). We'd be much smarter in a richer environment, too.

https://intelligence.org/2017/12/06/chollet/

[barrkel]

There's a parallel between something being logical and it "feeling right" without a necessary connection at the "implementation level" between the two, just like there may be a parallel between an artificial NN recognizer recognizing something unambiguously and not caught awkwardly with multiple weak or conflicting activations, and a logical system using rules to determine a contradiction, without ever needing to embed the second in the first, however deep - it's just that illogical inputs didn't get good training because they either don't happen or have no meaningful training data.

I, personally, just know I don't use logical rules very often at all. Usually I apply them retroactively as a post-hoc justification, or narrative, to explain a sense of discomfort or internal conflict or dissonance, but I have no way of knowing if my rationale is true other than how it makes me feel - I'm simply relying on the same mechanism, with an extra set of pattern recognition learned specifically to identify fallacies and incorrect logical constructs. If I didn't have that extra training, my explanations could be illogical and I'd be none the wiser.

I think humans are very bad at logical reasoning and very inefficient at it. Only a small % of the population ever does it and they usually do it incorrectly with biases, constructively to justify an already held conclusion. They're great at pattern recognition though. I don't think logical reasoning is anywhere on the critical path to human level AGI at a deep level. It could very well be a parallel system though to help train recognition if we don't figure out better ways of doing that.

[joe_the_user]

Well, neural nets and similar things laughably worse than AI systems when confronted with "real world" situation.

I wouldn't argue with the point that humans use rigorous logic and overt rules-based behavior much less than they imagine (your summary is very much a summary of the other-NLP model of mind, which I know).

I'd argue that while "refined" logic, systematic logic, might be rare, fairly crude logic, more or less indistinguishable from simply using language, is everywhere and it an incredibly powerful tool that human have. Again, being able to correct object recognition based on things people tell you is an incredibly powerful thing. You don't need a lot of full rationality for this but it gets you a lot. And that's just a small-ish example.

[Retric]

Intelegence is not limited to what Humans are good at. People are really bad at several tasks where current AI tech excels, but those things tend to be excluded from the conversation.

AGI that is as smart as say a rat would easily qualify as AGI even without language skills.

[joe_the_user]

Intelligence is not limited to what Humans are good at.

Being able to implement all the things human are good at, however, should be able to get us everything that we could do, because anything we could create, it could create too.

AGI that is as smart as say a rat would easily qualify as AGI even without language skills.

Indeed, but while a full language-using AI is ways a way at least, using language is one thing that's at sort-of describable/comprehensible as a goal. A rat is a lot more robust than any human made robot but how? Overall, I keep hearing these "there's intelligence that's totally unlike what we conceive" argument but it seems like computer programs as they exist now either do what a human could do rationally and more quickly (a conventional program) or heuristic duplicate human surface behavior (neural nets). You could sort-of argue for more but it's a bit tenuous. Human behavior is very flexible already (that's the point, right). And assuming AI is hard to create, creating something who properties we to-some-extent understand is more like than creating the wild unknown AI.

Also, "Getting to rat level" might not be the useful path to AGI. If we simply created a rat like thing, we might win the prize of "real AGI" but it would be far less useful than something we could tell what to do the way we tell humans what to do.

[visarga]

A rat can do something else that a neural net can't - it is a self replicator. Our neural nets don't have self replication or a huge, complex environment and timescale to evolve in. Self replication creates an internal goal for agents: survival. This drives learning. Instead, we just train agents with human-made rewards signals. Even a simple environment, like the Go board, when used for training many generations of agents in self-play, easily leads to super-human intelligence. We don't have the equivalent simulator for the real world environment or care to let loose billions of self replicating AI agents in the real world.

[joycian]

Survival is instrumental to any goal. Not only self replication would create that drive.