[middayc]

But didn't LLMs read all the math books and can't really do arithmetics (they need special modes / hacks / python to do it I think)?

So why would they be able to "read" the docs and use that knowledge except up to pattern matching level. That's why I also assume, that tons of examples with results would do better than lang docs, but I haven't tested it yet.

[vukgr]

While I don't like to argue for LLM competency, you have to remember that at the end of the day LLMs are word generators. They will always be bad at math unless there is a major structural change.

So while they cant learn arithmetic they should be able to learn programming languages given that they are way closer to what it was designed and trained for.