By this logic prostitution is no different than any economic transaction. Goods are made by people who are virtually, if not in fact, slaves. Does that mean you refuse to buy anything you didn't make with your own hands?
It's not realistic, though. You can't possibly research the povenance of every garment, food item, piece of electronics, drop of oil, shelter, and building material you buy. It's impossible.