Negative externalities play a large factor in the cognitive split. I'm more willing to tolerate a minuscule amount of people who hurt themselves more than a minuscule amount of people who hurt a random innocent.
At least here in LA, the drug problem has reduced many to husks who assault regular pedestrians on a daily basis. I understand your point, but drugs has many external factors besides ruining the user's life.