[jimfleming]

Highway connections are actually used heavily in NLP in combination with LSTMs.

[slashcom]

And replaced by residual connections in transformers, which are absolutely dominating LSTMs now.

[stochastic_monk]

Transformer-XL uses recurrence, and most NLP SOTA is still with LSTMs. I’m not sure I’d expect attention mechanisms to fully replace recurrence.