Unfortunately you still need the cartels to get the drugs into the US which is where they make most of their money anyways. I imagine parcels from Colombia, Peru, or Afghanistan are not likely to fly under the radar.
How so? I'm not an expert, but it seems most of the violence is concentrated in Mexico near the border as cartels battle for prime smuggling territory.
Even us not being experts. Can you honestly say you haven't heard of the destructive and violent effects of drug cartels in the US? Gang wars, turf wars, incarceration due to drug possession, etc.
I don't know about 'most drugs on silk road' but I've never heard of people cultivating coca, poppies, or sassafras trees for drugs in the United States. That's three major drugs right there (cocaine, heroin, ecstasy).