[calf]

Couldn't memoryless neural networks still possibly learn the Next-State function of a Finite-State machine? Depending on the training algorithm. Especially if the eventual usage of such networks is to be called over and over again to generate the next token; conceptually this to me seems analogous to the process of finitely unrolling a while loop or a computer pipeline.