Hacker News new | ask | show | jobs
by SkyMarshal 1107 days ago
What is your sense on how well current AI models can handle mathematics, vs human language? Is hallucinating or inventing info a problem with math too? Do you guys do anything to minimize, mitigate, or solve it?

And do you think current AI models will be able to solve as yet unsolved problems in math and physics, perhaps with increasingly large datasets or augments like Trees of Thought or process supervision? Or do you think it will require a different fundamental model?
2 comments
easygenes 1106 days ago
As someone who has been using GPT-4 and Claude working with Jupyter notebooks the last month, I’ll say that sometimes I am surprised by how well they can understand and segment out a problem. Other times it is obvious I’m asking too much of them, because no amount of clever reframing gets them to a right answer.

A surprise win was as much luck I had talking through an issue and getting correct, functional one-shot time grid and 3D plots including extra complications for the calculation of the axes.

A recent lose was trying to get it to generate a script that could optimize the gravity coefficient in the Reddit algorithm based on target timeframes and historic data. It offered salvageable output, if you pieced together several attempts and reframes, but never pulled it over the finish line.

link
visarga 1107 days ago
Not even humans with PhD can do physics research without a lab. How could a model with just a few tokens of context window and no body do that?
link
ElFitz 1107 days ago
What would should have a lab have, today, in the context of research on topics like string theory? That a simple office with a desk would not have, I mean.

Genuinely curious.

link
Zondartul 1107 days ago
Most of researchers' work is with formulas and reading articles. Even if you had a lab, you'd still need to think long and hard about what exactly you are trying to measure, and then design an experiment on paper before it can be carried out.

So it's possible, in theory, that an AI-in-a-box could do the hardest parts of science aside from experiments. Whether modern LLMs are any good at it is a different question.

link
SkyMarshal 1106 days ago
Obviously I’m talking about mathematical/computational physics, not experimental physics. I don’t expect the AI to build itself a tokamak to run experiments on.
link