[lukebuehler]

It’s endlessly fascinating to read the AI transcript of an expert who _really_ knows how to cut to the chase. It just shows how much you can potentially squeeze out of these models. I’m also surprised to see that even Terrence Tao seems to use it in a way that resembles, in progression, how I use llms in my area of expertise (emphasis on progression and usage patterns, not absolute skill, obv I don’t match that): short pointed questions that goes all in on the jargon and machinery of the field and steers the llm hard (eg no softballs). I’ve noticed that llms switch their tone and meet you basically more or less on your level.

[michaelchisari]

It reinforces how to "learn AI" is to first master the problem domain.

I can use AI for coding after decades of coding. I can't use it for theoretical physics because I can't evaluate the responses.

[larsfaye]

Correctamundo. That's EXACTLY what I just wrote about. And how we'll create the next generation of people who know how to evaluate responses when we're creating a generation of people who are increasingly reliant on LLMs to do the work makes for a strange paradox.

https://larsfaye.com/articles/ai-coding-will-prevent-experti...

[jeef_berky]

It's so obvious to some and may never be considered by others.

If someone asks about AI I tell them step 1 is ask it how to do something you know all about. Step 2 is consider everything else you ask will be that inaccurate.

[codebolt]

As a dev with three decades experience writing code, I use it for stuff I know all about all the time. And these days, the majority of the time it is more correct than I would've been without a few iterations of trial and error.

[OGWhales]

Gell-Mann Amnesia effect

[rnd0]

Honestly that seems kind of pointless and a waste of time. Might as well work it out by hand if you're going to go through all that effort

[Kon5ole]

Good writeup, and certainly food for thought.

I wonder if this key point actually holds though:

>The skills to do so, however, are a function of someone who has experienced the friction and challenges over time that culminate in "good taste".

It's certainly sometimes true, but I don't think it's a general rule. Sometimes friction is just friction and sometimes you spend 1000 hours learning something that disappears and becomes obsolete or at least irrelevant to the goal.

Programmers used to need to know the instruction set of the CPU, assembly language and so on. Some still do but for most developers today that's not useful. Everything you know about 6800 assembly will not make your note-taking app any better.

I think we are in a state where AI tools so easily mimic what we used to do by hand that we think the friction is gone, but that's because we haven't raised the bar yet. One day we'll look at the Fable one-shot that's better than anything we ever made by hand ourselves and say "It could be even better".

And then the friction is back, just on a new level.

[smsm42]

> Everything you know about 6800 assembly will not make your note-taking app any better.

It could. It could make a simple note-taking app not take gigabytes of memory and take visible delay on each click. Most people don't bother of course because simple note-taking app is not worth the effort. It's possible to do better, it's just often not practical.

[Kon5ole]

>It could. It could make a simple note-taking app not take gigabytes of memory and take visible delay on each click.

But the taste that tells you a note taking app should be fast doesn't come from your knowledge of assembly. It comes from using the app.

Being able to have an AI generate 8 variants of an UI and 5 variants of a storage mechanism is more helpful to reach the goal of a good note-taker. Trying out those prototypes and tweaking them to perfection is also friction, only it happens closer to your actual goal than doing quicksort in assembly.

There are of course examples where the friction does help, and where the "aid" of the tool deteriorates useful skills, but I think that will sort itself out over time. Useful skills will remain, useless ones will disappear, as they always have.

[danielezell]

Question from the peanut gallery here. Why not just train an LLM to write in Assembly? Or bare metal? Validate by compiling. Or binary, no compilation needed. Can't a frontier LLM find the bugs better than we? As long as the program is built from small, verifiable subunits, the pragmatic problem of creating an efficient program is in every careful planner's reach. Seems the friction is nearly gone. I have no skin in the game here, but this is a developing (no pun intended) expectation for me.

[chorizo]

> Everything you know about 6800 assembly will not make your note-taking app any better.

For most apps, it makes sense to profile your app for expected use cases. Put even minimal thought into making the hot codepaths faster by moving unnecessary operations out of it.

[pbronez]

Great article… been worrying about this one quite a bit:

> If these tools demand expertise, yet the tools can actively circumvent the friction that cultivates expertise, then what is the path for one to become an expert so they can effectively use these tools?

[rkuodys]

I am not sure if I'm ignorant or have too much self esteem, but in cases where domain is not my expertise I try to cross challenge using different models to get less-biased and hopefully more correct insights.

[beng-nl]

I’m not trying to be snarky, but.. I wouldn’t try to upgrade an answer from a stranger you can’t fully trust to an expert answer you can trust by asking more strangers who have all only read the same books and use the same system for summarizing the books.

Similarly, sometimes I have to manually validate the output of an ai tool, analyzed over a large text I can’t practically read and understand, and best way I’ve found is to ask the tool (or another ai) to ‘show your work,’ ie make it help me make the determination by showing places in the text I have to read to follow its reasoning. We can never trust another ai directly to assess the validity of another ai.

[gamerDude]

This is some work that I want to see in education. How do you validate if something is true or not.

A lot of times that will take physical tests. Or in the case of math/logic, tests to validate each line or validated sources of previously proved theorems

[RugnirViking]

You use the scientific method. Make a guess about how the system works, do experiments aimed to show that's not the case, adjust guess accordingly, and repeat

[jambalaya8]

This works only when the inputs and conditions are all the same. In the real world, that isn't really possible anymore for a lot of things, like data sciences operating within a large pulsating mass of live net.

[satvikpendem]

The paradox is analogous to AI poisoning its own training data as more and more AI generated content is released on the Internet. Indeed, I see a hard terminus for both man and machine at some point.

[dnemmers]

Absolutely right. First gen models are trained on 'virgin' non-LLM output. Subsequent models are tainted by ingesting AI replies. Rinse and repeat, and all you end up with is AI copies of AI replies, and the noise will completely overtake the signal.

[literalAardvark]

I'd have been more worried about that had the models not gotten incredibly good over time.

But they did get good and this seems like a non-issue.

[PeterStuer]

Fully agree but ...

Industry reality is that for bespoke software solutions we have been running for decades on non-technical people straight out of a 5 day "boot camp" copy/pasting together "solutions" from SO, or "Sharepoint Configurators" cobbeling together a LoB process where is takes 3 minutes to get to the next screen with a 10% error rate etc.

[pertymcpert]

Well put, great article.

[flashblaze]

Such a good read!

[edot]

Nicely written!

[natsucks]

More and more the skill of being able to ask the right question seems critical to me, and I don't know how one can do that without deeper and deeper domain expertise.

[kurthr]

Asking the right questions has almost always been one of if not the most important skills. There are often an almost infinite list of possible approaches, but knowing the domain, you can get close to optimum very quickly. If you just rely on the LLM, it will give you the most common bog standard approach, including the usual bugs and usual quirks.

[amelius]

Yes, but sometimes the right question is "do I know enough about this subject to be asking questions about it?"

[corford]

Succinctly put by Pablo Picasso's remark: "Computers are useless. They can only give you answers".

[XCSme]

I agree. I (15+ years of coding) feel like I'm 100x more productive now using LLMs to code.

But I don't think a beginner would have the same experience. The AI still makes A LOT of stupid mistakes and decisions, but I catch them early enough (sometimes while it's still showing it's reasoning steps), stop the prompt, guide it on the right path, rinse and repeat.

Sometimes I am lazy and give the AI a broader prompt, let it do its thing, and then I come back to see that it spent 90% of the time working on some part/feature/implementation that was not really needed and that it over-engineered the solution.

I rarely write any line of code know or manually change any code, I tell the AI how to do it and what to watch out for. Many times it catches some edge-cases before I even haven to think about them. I do still feel like both me and the AI could miss some edge-cases now, because I'm thinking less about the implementation and what problems can arise, but I feel like 90% of "gotchas" are already engrained in my planning after so many years of coding and problem solving.

[vouaobrasil]

> I can use AI for coding after decades of coding. I can't use it for theoretical physics because I can't evaluate the responses.

That is what will happen though to future generations: they won't be able to use it for anything because none of them will have the "decades of coding" experience that you have had the privelege to have without AI.

[singularity2001]

They will have decades of experience with AI and they will be able to guide them by sniffing their hallucinations from single words.

[retrochameleon]

Hopefully in decades hallucinations will be largely solved.

[tempfile]

Hallucinations are an essential feature of the technology, they cannot be "solved". You may as well hope that we solve the halting problem.

[timacles]

its just as likely hallucinations will only get worse because their source data will be riddled with hallucinations

[eru]

You can't hallucinate a working lean proof.

[jstummbillig]

That is probably not going to be the important difference.

You can't use it for theoretical physics because you have no meaningful question and there is no result that you can do anything with.

In contrast you don't need to be a coder to understand if your to-do list for cats works: You have an idea of what you want. You know the rough shape of what an app is and what it can do. You can put it in front of your can and look at it go. Or not.

[tdeck]

There are still some interesting pitfalls though. A friend of mine vibe coded a small browser based app to track something personal over a long time period. Eventually he plans to share the app with others. It worked, but I was curious where it stored the data and he didn't know.

It turned out it was using LocalStorage, which is not entirely unreasonable, but there are obvious drawbacks (e.g if you move the file it might become inaccessible, if you switch machines there's no convenient way to transfer the data, there are all kinds of ways to lose it, etc..).

In this case "how is this storing my data" is a fundamental question with plenty of implications for your app over its lifetime, but most non-computer people don't think to ask it at all. These days many users enter CS programs without knowing how to manage files and folders on their computer, because even that is often abstracted away.

[victorbjorklund]

Depends on what you are building. Yes, a todo app is easy. What would the physics version be? Probably ask ”what are the acceleration of a failing object?”

But if you are building some complex data science statistical model and you don’t have any domain knowledge you won’t even know what to ask for.

[mkagenius]

> I can't use it for theoretical physics because I can't evaluate the responses.

I think pretty much the opposite. I can ask it to explain to me in ways I understand it. Even drill down the simplest of equations. Since llms have infinite patience. All I need to learn anything is patience.

[case]

I think the issue is that LLMs can be so confidently and convincingly wrong about anything.

And if we (LLM tool operators) don’t know the subject matter in question, we can’t easily distinguish what they are right or wrong about.

[dsego]

It's not just what they output, it's also what they omit. I've unfortunately seen this when doing research on medical issues, and it gives you a false sense of having an understanding. It never warned me to watch out for symptoms that weren't specifically discussed in the context, or that some symptoms could be a different underlying condition.

[budsniffer952]

While humans are infallible and correct.

[porridgeraisin]

Yes, when you use a standard theoretical physics textbook / course sequence, it is 100% correct.

More broadly, LLMs or anything at all, needs a verifier. If the task can be automatically verified, great, then anyone can use them. If you can't automate the verification, then you need to be able to verify it using your knowledge. Knowledge required to verify is lower than the knowledge required to create in very few cases. This is why you still need a fully trained human verifier. We are yet to reorganise the overall "tasks" in the economy such that verification can be done with much lesser knowledge, for no reason other than that there was never demand for this until creation became automatic few years ago. It is possible and is slowly being done, there are many many startups working on automating verification in different fields and in many cases we will see fields reorganise themselves to be more amenable to automatic verification. Note that _effort_ required to verify is much lower than what is required to create, for almost anybody, and LLMs have economic use just due to that alone, albeit in the hands of a knowledgeable human.

[lanstin]

Math professors are remarkably good at answering math questions or solving ad hoc math puzzles correctly. Also at stepping back and asking, why are you trying to show this result, what do you think it has to do with the thing we are trying to prove. Much better at knowing when the student has gone off the path in some subtle way than the LLM (without a Prof Tao in the chat).

Professor Tao also put out some YouTubes of him working with an older LLM to do Lean proofs, and his intelligence matters - things where I would be stuck for hours trying to understand what was failing in the model proof were just instantly clear to him and fixed in thirty seconds.

And I am an ok coder (rather than a bad maths grad student), but the LLM will happily thrash around the edges of a problem with me with no clear convergence when I don’t have that clear insight and the problem is weirdly presented enough; I still find the trick to walk around the block and disengage and then return knowing exactly what to do (now prompting it to the right thing) to be a super power for getting what I want out of the coding system.

[mkagenius]

They are solving jacobian conjectures, I don't think they will hallucinate till masters level of any subject.

Edit: do give counter examples if you have any in maths, physics, chemistry, biology etc

[baq]

Both fable and sol are confidently wrong all the time in their most cherished domain - software engineering - I can’t quote anything because they’re working on my employer’s code bases. They’re much less wrong than their predecessors and they’re also quite good at point out their mistakes, but they’re still wrong a lot.

[mkagenius]

they are wrong on the specific nuance of my codebase too, i am talking about learning something - i can still learn everything about software engineering talking to a bot.

[seanhunter]

Here you go https://claude.ai/share/c8407777-1ccf-402b-9010-7bc57228b943

I asked Opus 4.8 to critique my algebra notes (these are definitely not masters level- just undergrad second year). It hallucinated an error it claimed I made in the notes and then put in a correction I didn't need because what I had written was correct.

What I said in my notes was:

   Notice that a cyclic group is a degenerate (in the sense of "smallest
   non-trivial") case of a finitely generated group where the generating set is
   a singleton.

It left-off the "non-trivial" and said that what I said was this was the smallest case of a finitely-generated group which is incorrect because it excludes the trivial group.

The point is I see the LLMs as a "smart friend"/colleague I can work with but I do think critically about what I get told and don't just take it as face value because it's not always correct for sure even in relatively basic cases like this.

[tempfile]

This deduction is baseless, there is no reason to think an LLM will only hallucinate on complex topics. They still get the "number of es in seventeen" question wrong regularly. The kind of mistakes an LLM makes has no clear resemblance to the kind of mistakes a human makes, because they are not doing the same thing.

[mkagenius]

Specific quirk of llms, also pretty much solved by existing thinking models.

I think you guys are misunderstanding me, you can still talk to a llm to learn everything about physics or maths.

[BoxOfRain]

One area that LLMs tend to do badly in still is sailing. I sail casually and I have friends who are instructors, basically all the LLMs we tried gave really unsafe advice for a particular manoeuvre. Claude was the notable exception getting it mostly right, but even so I wouldn't rely on it there.

[radlad]

This is blatantly false based on my usage of Fable and Opus. Though they are much better than they used to be.

[theplumber]

Indeed that’s the case but most of the programming problems are solved problems and unless you care how things are done someone with less/no knowledge can still make great use of programming to do what it needs to do.

Let’s not pretend you need to be programmer to use AI for programming like mathematicians need to solve math problems. Programming (most of the time)solves real problems rather than abstract constructs.

AI is already good enough to create the next “Facebook”(v1 and maybe v2 as well) without any real programmer. This makes the AI a big enabler and reduces the need for programmers vastly in the early stages of any business.

That being said as more online businesses will flourish these will require actual programmers after they get enough traction so the debate is still on if it will lead to massive layoffs in our industry.

[lanstin]

That seems only partially true. I have no doubt it could crank out a UI like Facebook, but having it work for high volume is less well represented in the training data. Things that seems obvious to us, like sharding, using asynchronous messaging systems to provide more durability at the cost of instant consistency, traffic shedding, intelligent clients, conscious decisions on CAP properties in the light of user experience. etc., are not going to be requested by that non-technical founder.

Facebooks early years were dominated by a concern for users per engineer ratio; the rather florid style of LLMs suggest that they will generate so many systems of such a high complexity you will get Hadoop levels of non-application support needed - forget self healing, it’ll need constant LLM spend just to keep running at scale.

[theplumber]

I call it BS. When you are hitting real scalability issues that the AI can no longer help you are already a successful business. The non technical founder will not know about async message passing but will ask the AI why the site is loading so slowly and sure enough the AI is able to implement caching and various mechanism to improve the performance and observability. You are hitting a wall when you do novel stuff (I.e a special search engine) but something like Facebook which is more or less a CRUD app is not a problem at all. And this is the case with Today’s AI systems(GPT Sol, Kimi K3 etc).

I can’t imagine what models will be able to do next year, especially the open weights ones that are not nerfed for economic or other reasons(I.e Fable saga)

[RugnirViking]

Have you ever tried to get an LLM agent to improve performance in a complex system? It's not good at it at all, it doesn't have a very scientific mindset about it, loves to do premature optimisation and "fancy tricks" while ignoring/missing glaring issues/gains - I'm sure it could be taught to be more methodical (verify what is actually slow before attempting a fix) but it's anything but right now.

Even if you walk it through the process, it also has terrible intuition about what time certain things "should" take - dismissing the possibility of massive speedups because it thinks some number is normal etc

[lanstin]

I thought the model was the one person company with enough usage to be worth 1 billion USD. Badly designed systems can hit bottle necks at like a few thousand DB calls per second which a well designed system can handle on one container. Getting those new features adding to table width or SQL length (or document nesting if you are going that route) with each new feature, I feel the company with a few experts will be at an advantage. Like the old beating the averages essay on using lisp. More smart flexibility wins in the marketplace.

[bdcravens]

True, but I think it's the difference between doing it for a single $200 subscription vs spending $200 a day in tokens.

[theplumber]

I think 5 x max subscriptions would suffice.

[ignoramous]

> I can't use it for theoretical physics because I can't evaluate the responses.

A tool, even if it is a chisel, in the hands of a master sculptor would obviously result in a wildly different outcome.

[amelius]

But the AI can teach you to do theoretical physics, by guiding you through materials it selected for you, helping you with questions, etc.

[mrguyorama]

Being good at "generating text" doesn't make you a good teacher. It was trained on internet users, not trained educators.

Most people posting on the internet, especially people who know about more complex subjects, are terrible teachers. Teaching is it's own skill

[amelius]

AI certainly does not seem like "the average internet user" to me, which is what it would be if your comment was anywhere near a good explanation for how it works.

[DiscourseFan]

Yes but, AI can still be used for relatively common applications (coding basic pages), and furthermore you can build agentic loops to check more difficult code, or explain the conceptual order of the code, and I imagine over time the domains which will be considered “basic” for the AI will continue to grow, but perhaps so will the abilities of its users…

[nathan_compton]

Yeah. I am a professor and I see a lot of other professors saying they are pivoting to "Teaching AI". I tell them I am pivoting harder to teaching fundamentals because using AI is easy. Its the knowledge which is hard.

[tty456]

But someone who is not well versed in coding can evaluate the output simply by running the built app and testing it, which is an oversimplification and omits the tech debt and eventual issues that an experienced developer would catch or restrict the output.

[budsniffer952]

Ah, yes, "experienced developers" see all the problems and resolve all the tech debt preemptively, which is why the state of software (even pre-LLM) was so great.

[jgalt212]

My rule of thumb is: Never ask an LLM a question where you don't know what the right answer or wrong answer looks like.

[inigyou]

I recently prompted ChatGPT about theoretical physics because I wanted some examples of AI slop. It affirmed my hypothesis that dark energy results from the nonzero VEV of the Higgs field. Wow! I'm so smart! I solved dark energy!

Any actual physicist would probably be able to tell me why that's a category error. I don't know why because I'm not one. But there are actual mathematics underlying a statement like that and I'm 99% sure the maths don't work like that.

By the way I think that's why everyone thinks of so many weird physics ideas more than other fields. It's because things are explained in words that hide math, and you can make hypotheses in words that would be obviously nonsense at the level of maths. Like your boss asking why you don't just recompile the cloud.

[mastermage]

Or in the words of Clickbait articles: MATHMATICIANS SHIVER OF THE IMPLICATIONS OF AI WILL MATHMATICIANS NO LONGER BE NEEDED.

to which betteridge's law of headlines says: NO.

[v9v]

That law only applies to headlines ending in a question mark.

[mastermage]

It is question i just foegot the Questionmark because i am dumb.

[wiether]

  > I’m also surprised to see that even Terrence Tao seems to use it in a way that resembles, in progression, how I use llms in my area of expertise

I didn't understood anything about the thread, but reading Terrence's messages was weird because it looked exactly like the discussions I have with LLMs

I've mostly seen people trying to oneshot a result, while I'll quickly experienced that going through steps/discovery was more effective and more satisfying, since you can always steer it back in the right direction; while oneshotting is hit (and it kind feel like magic) or miss (and you'll have to rework your prompt).

[neurahawk]

I was struck in the same way but I think it makes sense in terms of a thinking partner.

It is still ultimately Terence that is steering things.

What is crazy to me is how few of other people's conversations like this I have actually read.

Tao is really great for this because the anti-AI crowd can't really chime in and take the thread in a pointless direction. It is hard to think of another human alive who can carry the weight of unassailable authority in the same way.

[skeledrew]

I encountered something fairly similar working with Claude a few days ago. For a current project I've been fairly hand-wavy with requirements since I was getting good results, but it seemed to be failing hard on some key points, so I started to be more strict with it. Even after the fails were resolved, I've noticed that Claude now behaves differently within that project, carefully checking and rechecking things up front and also looking to me for guidance more often. Mildly irritating, but if it works...

[CamperBob2]

The most powerful way to use Fable in particular seems to be to pit it against Sol as a reviewer (or, I suppose, vice versa.) After going back and forth a half-dozen times, the result tends to be pretty much bulletproof.

What's somewhat disturbing is just how much Fable's code really does benefit from the review. It tends to leave a lot of low-hanging fruit, and you can see it getting kind of impatient when repeatedly called on it.

[apt-apt-apt-apt]

"We've now re-explored this question three times. While there were small differences along the way, one more would be pathological. Perhaps meditation or therapy would be more fitting."

This is basically how Fable told me to get therapy.

[Foobar8568]

The problem with that approach, the code becomes over-complex, over-engineered and more often than not, it will take a specific point and iterate over it.

It's still what I do 90% of the case until I feel it's good enough for my usage.

[divyekapoor]

What are you using to put Fable and Sol on each other?

[CamperBob2]

I just keep them in separate directories and use .bat files to copy files back and forth. Crude but effective enough.

1) Fable generates updated .c sources and .md design documents in myproj_fab

2) A batch file in myproj_sol copies the updated files from myproj_fab to myproj_sol

3) I tell Codex to "Review updated files, write findings to review.md"

4) Sol rips Fable a new one, usually

5) Another batch file copies review.md back to myproj_fab, where I tell Claude Code "See review.md"

I don't want to automate it any more than that, because I'll get lazy, stop watching the tennis match, and miss something important. Which will probably happen anyway...

[raz32dust]

The top comment has a counter example though https://x.com/DmitryRybin1/status/2079904005652893709

I think it's not even about the ability to steer the AI. Just the ability to ask the right questions

[eitally]

I am not impressed with that steering of the AI. In the case of Terry Tao, he clearly held a solid understanding of the problem domain and could also understand the outputs of the LLM at an expert level. He asked astute, topical questions that specifically probed areas of the problem that were either unclear or insufficiently specified, and ultimately ended up at a place where (presumably) he both understood the Jacobian Conjecture Counterexample exhaustively, and how it was derived. My impression is that through his learnings during the chat, he became sufficiently well-educated in this subdomain to apply some of these lessons to other mathematical pursuits.

In your counterexample, the human just repeatedly asked the model to "try harder". Not remotely the same.

[raz32dust]

I know but the point is that it worked too. You could argue one problem is harder than the other, I am not and expert so I wouldn't know. But at least it's clear the second result is also impressive, and didn't require much expertise. So it's wrong to say that you need expertise to get hard and impressive things done with AI.

[jv22222]

Came to say the same thing. Experts in any field have a huge unlock from AI.

[ooloncoloophid]

Is there a site with other good AI conversations like this?

[s0rce]

Yes, the AI models are so useful in areas where you are an expert and can guide it in a useful way. I use multiple chats delay to help figure out client projects.

[MeteorMarc]

I think the questions are sufficiently suggestive to induce hallucinations. Still a lot of manual work to check all these claims.

[lanstin]

For Professor Tao, I think that step is reading thru it once. Which is why the tool is more powerful for him than for us.

[ryangittins]

Well said! After a few years of LLM sycophancy convincing laypeople they've managed to make progress on or fully crack famous math problems it's satisfying to see a world-class mathematician really put it through its paces. It's all the more impressive that Tao seems satisfied with the conversation.

[conception]

One of the prompting guides I came across early on was to ask your questions using vernacular of the domain not lay terms because the correct data you are looking for will be more likely to be around/near the expert terms rather than in the pop/lay/newbie terms next token wise.

[tdu01]

Pretty spot on