Key point here, perhaps: People who don't eat x for religious reasons don't get this hate, at least not so that I've heard it. Jains may be weird but not hated, right?
I suppose what I'm saying is that lots people accept religious choice without thinking closely. If your choice isn't religiously motivated, they may take a look at the factory-made gloop you eat, notice that it doesn't look all that different from the factory-made gloop on the other shelf in the supermarket, and conclude (rightly or not) that your moral choice is a shallow one and despise you for your shallowness.
It doesn't matter whether you're actually shallow or thoughtful. IMO it doesn't even matter whether you eat factory-made gloop, so long as the supermarkets stock shelves full of plastic-wrapped stuff labelled "vegan" and you might be assumed to buy and eat it.