You can be held responsible. You don't have a right to obtain a substance just because you want it. It doesn't matter if it "goes in your body" or not. That's not how rights work. You have to justify the value to society.
It's exactly how rights work. Freedom of speech doesn't have to justify it's value to society, it's an innate right that's not up for vote. Google the founding fathers and natural law.
If freedom of speech didn't have to justify its value, you wouldn't need legislators to enact it as a law. Legislators aren't going to legislate without justification, and they shouldn't.