[notahacker]

I agree that my previous work experience can be viewed as "blue collar". I hadn't considered it may be difficult for a hiring manager to overcome.

It's not the fact that you've done "blue collar" work in the past that's the problem; it's the way you introduced yourself as the guy worried about his inexperience who did "blue collar" work until it became impossible. You should be introducing yourself as a self-taught developer with an unconventional background rooted in hard work.