[Patryk]

This same thing (i.e., using recurrent neural networks to predict characters (and even words)) was done by Elman in 1990 in a paper called "Finding Structure in Time"[1]. In that paper, Elman goes several steps further and carries out some analysis to show what kind of information the recurrent neural network maintains about the on-going inputs.

It's an excellent read for anyone interested in learning about recurrent neural networks.

[1] http://crl.ucsd.edu/~elman/Papers/fsit.pdf

[sushirain]

It's amazing how much was already known decades ago. Elman and others did much more, and hopefully, now the field will take the next step (which was long delayed), with the help of today's computer power.