[tovej]

This comment would be more useful if you actually provided those examples.

I'm also a researcher and agree wholeheartedly with the article. LLMs can maybe help you sift through existing literature or help with creative writing, at most they can be used or background research in hypothesis generation by finding pairs of related terms in the literature which can be put together into a network of relationships. They can help with a few tasks suitable for an undergrad research assistant.

[viraptor]

https://arxiv.org/abs/2409.04109

> we obtain the first statistically significant conclusion on current LLM capabilities for research ideation: we find LLM-generated ideas are judged as more novel (p < 0.05) than human expert ideas while being judged slightly weaker on feasibility.

It's a bit better than just finding related pairs. And that's with sonnet 3.5 which is basically ancient at this point.

[tovej]

This paper centers "novelty" but also finds that human ideas are more feasible, and that LLM-generated ideas are not diverse and that LLMs cannot reliably evaluate ideas. None of the ideas were actually evaluated by performing experiments either.

Pretty much what I would expect. The paper also seems to be doing exactly what I described, I don't understand how the technique is better than that?

[JimDabell]

> I'm also a researcher and agree wholeheartedly with the article.

The article says:

> Yet, every time I tried to get LLMs to perform novel research, they fail because they don’t have access to existing literature on the topic.

You say:

> LLMs can maybe help you sift through existing literature

> they can be used or background research in hypothesis generation by finding pairs of related terms in the literature

As far as I can see, these two positions are mutually exclusive. Aren’t you disagreeing with the article?

[tovej]

No, helping with chores is not the same as "performing research", it has limited utility for minor tasks, it is not essential and it does not even necessarily have a positive productivity impact. To illustrate the point, when I use vim to write LaTeX, would you say that vim is "performing research"?

[JimDabell]

The relevant part of what I quoted was not “performing research” it was “they don’t have access to existing literature”.

[tovej]

That part we also agree on, generating hypotheses is done on the basis of existing literature, lit review (& related reasoning) is done on the basis of existing literature. The blog author is talking about reasoning in the context of a new research topic.