There is a chain of violence that goes back from most street drugs. There is no victimless usage of drugs - you are directly damaging a lot of the US and almost all of Latin America at this point.
While you did qualify “street drugs” in sentence #1, you didn’t in #2, and so I have to disagree that the stinky plants in my garden are damaging to anyone :)