[riku_iki]

It's actually interesting results in a sense we see the limitation of LLM to memorize complicated information correctly. Gemini ultra also reported around 50% accuracy

[Davidzheng]

I think the SOTA is GPT4+tool use? I heard near 80%

[riku_iki]

Yes, tools help to advance over LLM limitations. GPT4 without tools is about 50% too.