[jruohonen]

> For starters I would like to have a chatbot who knows the scientific literature in a certain field, like climate science.

Mission impossible as most of the relevant literature is paywall'ed.

> Then I would like to be able to ask questions which are answered with pure scientific facts only, just based on scientific literature.

Mission impossible because LLMs cannot evaluate facts. Nor can they do source criticism.

[ChrisMartin33]

Couldn't one - pay the science mags for access - ask the LLM for the conclusions of the papers, ask it to throw out vague conclusions and deliver only what seems to be a hard proven fact shown in the paper.

[james-revisoai]

Have a look at "Consensus" App.

[gus_massa]

@GP: most paywall are weak.

Anyway, reading a paper is quite difficult. Some results expresed like facts in the abstract or conclusions are not fully suported in the data. You should always read the full paper. For now, it's difficult for an AI to distinguish facts from bulshit.

(I remember a few physics paper that prove strange things, but when you reed the proof it has an aditional hypothesis in the middle of the proof. The hypothesis makes sense in the physics context, but in math all the hypothesis must be in the text of the theorem.)

[ChrisMartin33]

You are right of course, every scientific paper, even peer reviewed, even published in the "best" journals, can be total BS.

I thought about just analyzing the conclusions. And to pick out whether they write "one might draw the conclusion" or "we have proven that...." I think that could be useful, although I am aware that in reality they might still have proven nothing.

[gus_massa]

In my area, I always ask who is the author team. There are a few and many are well known.

If it's team A, I'm almost sure it's correct. But I check anyway.

If it's team B then "Press X To Doubt". It's probably not wrong, but perhaps oversold.