The biggest problem is re-using passwords. Remember when that LinkedIn password database with plaintext/md5 (easily brute-forced) was leaked a few years back? And who knows what's going on with Heroku right now. There's been dozens, hundreds, thousands of leaks like this, from well known sites like LinkedIn to that small independent webshop where you ordered something a few years ago. And for at least some people those credentials that worked on LinkedIn may also work on GitHub.
Having a unique password per service solves that particular issue. But you're right that there's a "price" in that losing access to your passwords would leave you screwed. I'd strongly recommend making sure you at least memorize your email password, as well as backing it up in several places.