[alchemist1e9]

> That said, I think the fundamental problem of such tools is unsolvable: Out of all possible analytical designs, they create boring existing results at best, and wrong results (i.e. missing confounders, misunderstanding context ...) as the worst outcome. They also pollute science with harmful findings that lack meaning in the context of a field.

This doesn't seem correct to me at all. If new data is provided and the LLM is simply an advanced tool that applies known analysis techniques to the data, then why would they create “boring existing results”?

I don’t see why systems using an advanced methodology should not produce novel and new results when provided new data.

There is a lot of reactionary or even luddite responses to the direction we are headed with LLMs.

[uniqueuid]

Sorry but I think we have very different perspectives here.

I assume you mean that LLMs can generate new insights in the sense of producing plausible results from new data or in the sense of producing plausible but previously unknown results from old data.

Both these things are definitely possible, but they are not necessarily (and in fact often not) good science.

Insights in science are not rare. There are trillions of plausible insights, and all can be backed by data. The real problem is the reverse: Finding a meaningful and useful finding in a sea of billion other ones.

LLMs learn from past data, and that means they will have more support for "boring", i.e. conventional hypotheses, which have precedent in training material. So I assume that while they can come up with novel hypotheses and results, these results will probably tend to conform to a (statistically defined) paradigm of past findings.

When they produce novel hypotheses or findings, it is unlikely that they will create genuinely meaningful AND true insights. Because if you randomly generate new ideas, almost all of them are wrong (see the papers I linked).

So in essence, LLMs should have a hard time doing real science, because real science is the complex task of finding unlikely, true, and interesting things.

[alchemist1e9]

Have you personally used LLMs within agent frameworks that apply CoT and OPA patterns or others from cognitive architecture theories?

I’d be surprised if you have used LLMs beyond the classic chat based linear interface that is commonly used and still have the opinions you do.

In my opinion, once you combine RAG and agent frameworks with raw observational input data they can absolutely do real reasoning, analysis, and create new insights that are meaningful and will be considered genuine new science. This project/group we are discussing have practically proven this with their replication examples. The reason this is possible is because the LLM is not just taught how to repeat information but it can actually reason and analyze at a human level and beyond when utilizing it’s capabilities within a well designed cognitive architecture using agents.