[mindcrime]

David Silver's course on Reinforcement Learning contains some good information on Markov processes. See Lecture #2 in particular.

https://www.youtube.com/playlist?list=PL7-jPKtc4r78-wCZcQn5I...