Transformers are being used in Deep RL for at least months.
Try these: https://scholar.google.com/scholar?q=transformer+deep+reinfo...