It's not just what they output, it's also what they omit. I've unfortunately seen this when doing research on medical issues, and it gives you a false sense of having an understanding. It never warned me to watch out for symptoms that weren't specifically discussed in the context, or that some symptoms could be a different underlying condition.
Yes, when you use a standard theoretical physics textbook / course sequence, it is 100% correct.
More broadly, LLMs or anything at all, needs a verifier. If the task can be automatically verified, great, then anyone can use them. If you can't automate the verification, then you need to be able to verify it using your knowledge. Knowledge required to verify is lower than the knowledge required to create in very few cases. This is why you still need a fully trained human verifier. We are yet to reorganise the overall "tasks" in the economy such that verification can be done with much lesser knowledge, for no reason other than that there was never demand for this until creation became automatic few years ago. It is possible and is slowly being done, there are many many startups working on automating verification in different fields and in many cases we will see fields reorganise themselves to be more amenable to automatic verification. Note that _effort_ required to verify is much lower than what is required to create, for almost anybody, and LLMs have economic use just due to that alone, albeit in the hands of a knowledgeable human.
Math professors are remarkably good at answering math questions or solving ad hoc math puzzles correctly. Also at stepping back and asking, why are you trying to show this result, what do you think it has to do with the thing we are trying to prove. Much better at knowing when the student has gone off the path in some subtle way than the LLM (without a Prof Tao in the chat).
Professor Tao also put out some YouTubes of him working with an older LLM to do Lean proofs, and his intelligence matters - things where I would be stuck for hours trying to understand what was failing in the model proof were just instantly clear to him and fixed in thirty seconds.
And I am an ok coder (rather than a bad maths grad student), but the LLM will happily thrash around the edges of a problem with me with no clear convergence when I don’t have that clear insight and the problem is weirdly presented enough; I still find the trick to walk around the block and disengage and then return knowing exactly what to do (now prompting it to the right thing) to be a super power for getting what I want out of the coding system.
Both fable and sol are confidently wrong all the time in their most cherished domain - software engineering - I can’t quote anything because they’re working on my employer’s code bases. They’re much less wrong than their predecessors and they’re also quite good at point out their mistakes, but they’re still wrong a lot.
they are wrong on the specific nuance of my codebase too, i am talking about learning something - i can still learn everything about software engineering talking to a bot.
I asked Opus 4.8 to critique my algebra notes (these are definitely not masters level- just undergrad second year). It hallucinated an error it claimed I made in the notes and then put in a correction I didn't need because what I had written was correct.
What I said in my notes was:
Notice that a cyclic group is a degenerate (in the sense of "smallest
non-trivial") case of a finitely generated group where the generating set is
a singleton.
It left-off the "non-trivial" and said that what I said was this was the smallest case of a finitely-generated group which is incorrect because it excludes the trivial group.
The point is I see the LLMs as a "smart friend"/colleague I can work with but I do think critically about what I get told and don't just take it as face value because it's not always correct for sure even in relatively basic cases like this.
This deduction is baseless, there is no reason to think an LLM will only hallucinate on complex topics. They still get the "number of es in seventeen" question wrong regularly. The kind of mistakes an LLM makes has no clear resemblance to the kind of mistakes a human makes, because they are not doing the same thing.
And for the harder maths you want something not just to explain the answer but to diagnose your conceptual error behind your questions. Asking the teacher after class or tutorial type systems work efficiently for a reason.
One area that LLMs tend to do badly in still is sailing. I sail casually and I have friends who are instructors, basically all the LLMs we tried gave really unsafe advice for a particular manoeuvre. Claude was the notable exception getting it mostly right, but even so I wouldn't rely on it there.