Comprehensive benchmark of the most important (and recent) algorithms in RL. Extremely modular so that it's easy to implement your own algorithm and straight-forwardly compare its performance in many different continuous environments.
I am a happy user of it!