And yet, ChatGPT can generate these strings. Somehow despite using the wrong loss function it still seems to work by simply absorbing more training data.
It can also recognize them, albeit it tends to cheat by shelling out to python (which makes sense, since it tends to lose count on large strings just like a human...)
ChatGPT incorrectly put a space between the as and bs, but if we let that slide, there's still the issue that the best trained model in the article got 77.3% of the first 1500 strings correct, i.e. even if ChatGPT performed exactly the same, you'd expect it to get a single example correct more often than not.
The difference is that you told it the language you wanted to recognize. In the paper, they are trying to learn the language from example strings alone.