[kazinator]

Author doesn't seem to understand that LLM AI works by predicting tokens out of training data. The model writes a research summary because it digested academic papers and other sources in its training. When you say "AI can already do social science research better than most professors" that is false unless you mean the colloquial sense of "research" meaning "reading other people's existing stuff and paraphrasing it in my own words". But the AI doesn't even have "own words"; they are the training data's words.

If all scientists suddnenly do nothing all day but play with AI --- all research grinds to a halt!

[lubujackson]

Don't undersell AI - it also synthesizes and recombines those summaries in a purposeful way. Otherwise it couldn't product code that works in an existing codebase.

So it is able to process and act upon summaries and concepts. In other words, apply synthesis. What it can't do is understand what a useful result looks like without direction. So it could synthesize a billion pointless claims from source material, but we still need a human to know which ones matter (without a specialized framework to comprehend this). If you provide LLMs with an objective and source materials it is certainly capable of following threads of logic or building an argument backed by sources.

I understand the concerns about AI, but it is a powerful tool for discovery and synthesis.

[antonvs]

Another thing they’re often poor at is making an incorrect assumption and then going down a rabbit hole trying to unnecessarily solve for it. Without a discerning human in the loop, you can end up with large amounts of unnecessary output.

[in-silico]

This is true after pretraining, but reinforcement learning allows the model to discover strategies and ideas that weren't in its training corpus.

[antonvs]

Are you perhaps thinking of transfer learning, i.e. where training on one subject can be applied to another? RL is more about coercing models in particular directions.

[directorscut82]

This is not what RL does, and please stop anthropomorphizing statistical modelling as the model certainly does not discovers ideas.

[in-silico]

What does RL do then if not discover strategies and solutions that weren't in its training data?

[applfanboysbgon]

RL adjusts the learned probabilities to conform to a secondary source other than the raw training data, for example (but not exclusively) human feedback. Putting it in extremely simplified terms: If, owing to the training data, the learned probability for "green people are _" is 70% to be followed by "inferior", you may use RL to massage this, de-scoring it every time it produces "green people are inferior to red people" and up-scoring it every time it produces "green people are an ethnic group originating from Greenland". Doing this will adjust its learned probability for that sequence of tokens.

At most, RL can be described as injecting information from a secondary source. It is not extending a model's programming to do anything other than what it was already doing, probability-based token prediction. It simply alters the probabilities.

[in-silico]

What about things like AlphaZero and Atari gameplay, where the model has zero prior knowledge and learns superhuman ability purely using RL?

With sufficient RL sampling/training, there's no reason an LLM couldn't similarly develop entirely new skills, especially in verifiable domains like math and code.

> It simply alters the probabilities.

Yes? What else would a learning system do besides alter its behavior? (and you can just sample with argmax or pseudo-randomly of you think probabilities are a problem)

[antonvs]

Functionally, i.e. focusing only input and output, a model can certainly discover an idea. That’s not anthropomorphism.

Similarly, people often object to using words like “reasoning” and “understanding” in relation to models, but again, functionally, models observably demonstrate both of those qualities - you can test for them and measure their proficiency.

The fact that this discovery, training, and understanding is implemented in terms of a statistical model isn’t really relevant. If it were, you could similarly argue that humans don’t discover, reason, or understand, we just process chemical and electrical signals through our biological neural network.

[squidbeak]

What about Alphafold?

> But the AI doesn't even have "own words"; they are the training data's words.

If the AI understands those words, in what sense aren't they its 'own words'? Are you arguing that nothing but neologisms count?

[kazinator]

I would say that I don't consider that to be an LLM.

[Legend2440]

It's not literally an LLM because the L stands for language, and it's not trained on language.

But it is the same transformer architecture, and it is able to generate novel proteins in the same way that an LLM is able to generate novel sentences. AlphaFold 3 is a diffusion model, so it's most similar to the AI art generators.

[kazinator]

But it's used on the thing that it's trained for. LLMs are trained on language, but then used as a substitute for thinking, which it naively looks like they are doing due to the smooth language.

Protein folding is a kind of syntax. If you train on protein folding and then use it to obtain protein folding results, you are using a screwdriver to drive a screw: that checks out.

Nobody should be arguing along lines analogous to the claim that a good neural net trained on handwritten digits is not suitable for classifying handwritten digits.