[freyir]

You don't mention (a) your reason for pursuing a PhD / what you plan to do after grad school, (b) what you've achieved so far in the past 3 years, or (c) your field of study. Without at least that information, it would be very difficult for anybody to advise you here.

> I don't have many references because, I'm in graduate school.

That's not a valid reason. Networking takes work, and you should start now. I can't stress that enough.

> he won't leave a good word in for me for anyone who contacts him as a reference

If he's as bad as you say, his reference might not carry a lot of weight anyway. If you're set on pursuing an academic career, moving to a professor who does good science and has a good reputation might be worth another couple years in the long run.

> it's too late to switch groups without wasting an additional 3 years

In my experience, this isn't necessarily true. You may find an advisor who won't completely ignore the work you've already done (unless you have nothing to show for the last three years). Some people here advise you to meet with other professors and "explain the situation." Be extremely careful how you explain the situation, and don't disparage your current advisor.