[gchq-7703]

I'm fairly sure that these models don't work solely on the previous word, but instead are able to remember some level of information from history.

Otherwise, you'd reach a word like 'and' and couldn't possibly follow it with a logical statement that follows on from the previous part.

[ragebol]

This is why I said 'words', multiple :-).

My point being that these generation models should be conditioned on something more than just word history, like something they want/are instructed to express.