[rossdavidh]

I think these efforts point out something valuable, although probably not in the way the creators intended. Lots of people use "markers" of reliability, like citing your sources or making sentences with a certain kind of structure or tone, to estimate trustworthiness. These articles make it clear that it is entirely possible to have those markers, but be entirely incorrect in your assertions about the topic in question.

There is no particular reason to think that this is something only AI models do. Plenty of people do the same thing, working much harder at looking, sounding, and acting like a trustworthy source, without actually putting much work into knowing what they are talking about. I think the absurdly incompetent nature of some of these AI models, is a great illustration of that point.

[fullshark]

It took me an embarrassingly long time to understand that the previous marker of authority regarding news: "published in a newspaper" completely lost all its meaning as the blogosphere exploded, and publishing costs on the internet went to near zero. Kind of why I find the pearl clutching over substack hilarious, as if having a third party website sell ads on a writer's blogpost signals they are much more worthy of authority.

I think the air of authority these academic journals get is the next domino to fall. Get ready for a lot of "we used AI to write an academic paper and it got published in this journal" stories.

[trompetenaccoun]

>Get ready for a lot of "we used AI to write an academic paper and it got published in this journal" stories.

Already happened, and the linked example is far from the only case: https://www.nature.com/articles/d41586-021-01436-7

As someone who used to work in science, I feel the general public doesn't have much of an idea how flawed the peer-review system is in practice. Low quality journals that simply print anything aside, this was an issue long before such language models became good enough to write papers, because humans are perfectly capable of producing nonsense research without the aid of machines. I'm not sure what philosophies/religions will replace the current cult but ultimately it's probably a good thing that this blind belief in such institutions gets eroded. They should never had had that much power over people's minds to begin with.

[karp773]

I would argue quite the opposite.

In the world of nonsense and misinformation, competent and insightful sources become of supreme importance. In a sense, we find ourselves back in pre-Gutenberg times. The elite has access to the insider sources and knowledge while the masses have a hard time to find the truth in hearsay blogs, spam bot outputs, and memes.

The situation will hopefully improve when another gutenberg comes up with a novel information search algorithm.

[iudqnolq]

Yes, and the way this is corrected against is with reputation. Do that, and no one will trust you again. Seems to be working here.

Edit: A better way of putting this is that the risk of doing something is a combination of the odds of being caught and the consequences of being caught. It's much harder to catch a deliberately lying paper author than a mistaken one, so we make the punishment much higher to compensate.

[brookst]

“Apes don’t read philosophy.”

“Yes, they do, Otto. They just don’t understand it.”

[still_grokking]

https://m.media-amazon.com/images/I/81B1+kuYOaL._AC_SL1500_....

[cscurmudgeon]

> I think these efforts point out something valuable, although probably not in the way the creators intended. Lots of people use "markers" of reliability, like citing your sources or making sentences with a certain kind of structure or tone, to estimate trustworthiness. These articles make it clear that it is entirely possible to have those markers, but be entirely incorrect in your assertions about the topic in question.

Also, known as syntax vs semantics.

The bet in modern NLP is that syntax is enough to arrive at semantics.