[bkovacic]

Gemini Ultra should show good progress according to Google - it's supposed to perform better than 85% of computer science competitors, which requires a lot of logical reasoning. Lets see it once it goes live, but sounds promising.

[mtlb]

Their previous model was better than 46% of such competitors (according to them), so 85% seems achievable by throwing more compute resources at typical ML training. After all, training on millions of examples of logical reasoning will undoubtedly store logical rules in the model in some shape or form (it does so even in ChatGPT), yet the results are still more "convincing" rather than "correct", or "probably correct" at best, usually achieved with lots of postprocessing on top. GPT-4 is better than 90% of lawers at the bar exam, yet still manages to fail at reasoning on much simpler domains.

[mirekrusin]

Imaginary models hyped/faked/lipsticked by PR deparment, in future tense, in the field that advances on daily basis is quite weak in most discussions.