[zerobees]

I don't know this applies to you, but I've had several friends who convinced themselves they're exploring the frontiers of science, and it always turns out that their conversations spiraled into some sort of weird quantum-metaphysical gobbledygook.

LLMs are sycophants, and in long conversations, their sycophancy produces a positive feedback loop: the context window contains affirmations of incorrect interpretations / analogies, so the chatbot continues down that path because, well, that's the most likely completion of previous text. And before you know it, you're discovering the hidden fabric of the universe, which is always some Minkowski fractal spacetime tensor lattice manifold with subharmonic DNA nanotubes.

That is to say, unless you have a robust way to evaluate what you're learning, and to confirm that you're actually learning, I'd tread carefully.

[hodder]

You really just need to augment with tight prompting and know how to extract information and links to peer reviewed literature and well sourced information. Once again, technology here is a lever. Separating wheat from chaff has been key in academic and information pursuits forever and it is becoming ever more important.

[zerobees]

And you're gonna review that voluminous academic literature in a field you're not familiar with, right?...

That is my point: an LLM can be great if you know the field and can spot errors. Or, to a lesser extent, if you have some automatic feedback loop that the model can't easily game ("does this code pass unit tests?"). It's a lot less great if there's a risk that you won't detect the early drift.

[hodder]

Sifting through information to separate truth from fiction is a modern, experience-based skill that I’ve developed YES.

However, your perspective on how LLMs are used might be too narrow. While no one is suggesting using an AI to find a one-shot cure for Alzheimer's, LLMs are incredibly effective tools when paired with textbooks to master subjects like undergraduate physics.

[jplusequalt]

>LLMs are incredibly effective tools when paired with textbooks to master subjects like undergraduate physics

I agree that they can be helpful for explaining things if you're stuck. But since you used the example of physics, the majority of hard won physics knowledge comes from working lots of problems yourself.

Too many people are already used to LLMs circumventing the laborious aspects of things, so it takes a certain person to withhold from just having the LLM solve the problem for you when you're stuck.

Until we can figure out to teach people to fight against their instincts, then I don't think LLMs will lead to better long term outcomes.

[hodder]

On average yes this is almost certainly true. As I said above technology is a lever and most people are incredibly lazy. However if one has drive and determination they are also incredibly powerful tools

[meetingthrower]

100% you are so right. Don't rely on the LLM. It is a robot you give tools and context, and then you ask it to do something with that. It's amazing at that.

[jrflo]

If you’re cognizant of the sycophancy and not a total narcissist then it’s not too hard to figure out with a little experience when the models are just trying to hype you up