[twoodfin]

I think the key question is more effective at what?

I see plenty of anecdotal evidence that models have been trained fantastically well—and getting better—at writing to trigger the right neurons in the human population to produce “This is interesting/informative/correct” responses in bulk.

Could their ability to produce those responses run far ahead of their ability to actually achieve the last in reality? Sure seems plausible, and then where are we?

[s1artibartfast]

Many people are bad at writing, including scientists. Improvements could be articulating a concept in a way that the reader will understand it clearly.

[twoodfin]

My hypothesis is that an LLM’s ability to concoct prose that will convince even an expert of the validity of an idea is largely independent of the LLM’s ability to validate the idea itself, or whether the idea is correct in the first place. Especially if it’s being prompted to do the former, not the latter.

My anecdotal evidence is the LLM-generated, inchoate technical dross that is routinely upvoted onto the hn front page. Much of it isn’t even coherent enough to be wrong, but the readership here finds it interesting!

[s1artibartfast]

I agree prose and validation are different. I dont know anyone who contests this.