[vouaobrasil]

The true reason is that advice in the real-world is more than just words: truly good advice can only be rooted in the care and experience of another human being. And it's not just the care that the other person has for you, but the understanding that the advice has come from a place of experience of human suffering.

In terms of more technical questions like the one in the article, that's rather obvious too: ChatGPT is like the person giving advice about the stock market. It only speaks of averages that are already in the direction of the general movement of the masses. Truly good advice or ideas involves going against that momentum to find a new way.

Of course, AI will be helpful to some in some situations. But even that has larger societal implications: all the help it provides displaces the need for real human beings, and thus propels society into having even more problems, and thus it reduces a few current problems in exchange for more problems in the future.

AI does no ultimate good. It should be destroyed.

[visarga]

AI models can and do create their own experience, trillions of tokens per month of LLM output with human feedback, it scales a lot.

Besides what they learn from humans in chat rooms, they got feedback from code execution, search and other tools.

In general AI models embedded in larger systems can have feedback from outside. It is on policy data, like human advice.

[lm28469]

At some point you have to decide what you value in life, the originality of singular human experiences or the average aggregate of LLM users. Outside of purely technical problems ai will never be the answer (and in its current state it isn't even good at that)

Personally one interests me way more than the other, I'm not going to read a book about the average experience of an average spelunker, but I'll gladly read Michel Siffre's books.

https://en.wikipedia.org/wiki/Michel_Siffre

[Terr_]

> truly good advice can only be rooted in the care and experience of another human being.

Ideally, but advice can still be good/useful/accurate even when it isn't direct personalized love etc., or else nobody would ever make books, which are arguably deader than an LLM.

[fl7305]

> truly good advice can only be rooted in the care and experience of another human being. And it's not just the care that the other person has for you, but the understanding that the advice has come from a place of experience of human suffering.

Can another person give you truly good advice over email? Is it still possible if that person has never met you before, so that the advice is only based on what you have written to that person?

If yes, then it is possible in theory for an AI to give you exactly the same advice as this other human being.

How can you then say that email A from the AI is bad, while the exact same email B from a human is truly good?

> ChatGPT is like the person giving advice about the stock market. It only speaks of averages that are already in the direction of the general movement of the masses.

You have to distinguish between how AIs are trained (or rather, selected for) and whether that can lead to intelligence.

AIs are selected for how well they predict text.

Humans were naturally selected for how well they reproduce.

Neither one automatically leads to intelligence. But that doesn't mean that you have scientifically proven that either type of selection can never lead to intelligence.

A big LLM has on the order of 10^12 parameters. At the base level, it is not that different from a regular 10 TB hard drive. Even if each fact is only a byte big, you could't store more than 10^13 different facts before it starts running out of space. Your address space (or input vector) is only 13 digits long, or let's say 30 letters.

So you only have up to all combinations of 30 letters as the input vector, or about six words. The output vector (response / fact) is just one byte in this example. That is tiny compared to the length of the conversations and the amount of facts that today's LLMs can recall.

During training, the LLM evolves ever smarter compression to store as much information as possible. At some point, it becomes necessary to start developing some sort of model of the world and how different things interact to improve the compression even more.

Now, that doesn't mean that it can do this perfectly, or even well. It can be infurating to argue with an LLM once you hit the limits. But I do believe that we are starting to see the building blocks of real intelligence in the bigger LLMs.

> all the help it provides displaces the need for real human beings, and thus propels society into having even more problems

Your key claim here is a completely orthogonal question.

I would say that this question becomes more and more important the more intelligent the AIs become. Not the other way around.

An AI that can perfectly mimic how you would like to interact with other people would be living in a padded cell. All struggles and challenges that allow you to grow as a person disappear. We'd become like the people in the spaceship in Wall-E.

So you're doing your key claim no favors by trying to prove that the LLMs are dumb.

[vouaobrasil]

> Can another person give you truly good advice over email? Is it still possible if that person has never met you before, so that the advice is only based on what you have written to that person?

Intensionality is important: the fact that the advice came from a person and that we know another person is out there caring is important. Sorry to tell you, but we are not mere computers whose only functionality is dependent on direct input. Moreover, the more people care for each other, the better society becomes.

[fl7305]

> The more people care for each other, the better society becomes.

Sure, not arguing that.

> the fact that the advice came from a person and that we know another person is out there caring is important.

What if the AI pretends to be a real person? Does that not make you feel the same as when a real person responds in exactly the same way?

I'd agree that this does not make for a good society, but that's a different question from "how does the interaction make you feel".

I'm not saying we should have a society where people get much of their social interaction from AIs.

I'm saying that if you argue against that, you undermine your argument if you downplay how capable AIs can become.

In a sense, your argument is: "AIs can't make you feel the same way as another person can". But what happens if people start thinking "Hey! The AI makes me feel even better than when I interact with real people!" ? Your stated argument against AIs is gone.

[Devasta]

> How can you then say that email A from the AI is bad, while the exact same email B from a human is truly good?

Why are you asking us these questions when you could instead be asking ChatGPT?

[fl7305]

It's unnecessarily hostile and misguided responses like this that will push a lot of people towards interacting instead with always friendly AI-buddies.

But just like junk food and no exercise softens the body, interacting mostly with overly friendly AI-buddies will soften the mind.