[Calzifer]

I'm not a chess expert but I would expect there is a lot of information available about chess openings but not so much about midgame (it sometimes appears as if you just play a bunch of standard moves before the actual game begins).

One source even mentions that a chess opening consists of 10 to 15 moves which would match your observation that the AI gets worse after that.

Maybe the AI would appear more competent again in the endgame since I assume there is more training material for that again.

A bit surprising that it provides illegal moves since it should have enough training material not "understand" the rules but apparently prefers to produce sentences which appear like chess moves from the training material without any connection to the rules.