[jib]

3 years of anything is not worth a reference. Doesn't matter what.

You should figure out what is best for you, and do that. Without being you, I think it will be hard for people to give any advice other than don't worry about recommendations/perceptions. This isn't about that guy, it's about what you want to do.

Whether that is asking for a transfer, work on your relationship with the guy, stopping altogether or not, that's something you'll have to figure out, but it is about you, not about your adviser.