[bwfan123]

I had an aha moment recently. An excited AI researcher claimed, wow: claude could solve this IMO problem. Then, a mathematician pointed out a flaw which the AI researcher overlooked. The AI researcher then prompted the AI with the error and then the AI produced another proof he thought worked, but again was flawed. The AI played on the researcher's naivete.

Long story short, current AI is doing cargo-cult math - ie, going through the motions with mimicry. Experts can see through it, but excited AI hypesters are blind, and lap it up. Even alpha-geometry (with built-in theorem prover) is largely doing brute-force search of a limited axiomatized domain. This is not to say AI is not useful, just that the hype exceeds the actual.

[xanderlewis]

Everything you’ve said is obvious, and yet really needs saying and isn’t said enough.