The problem is "fair" is hard to measure, but "cheapest" is very easy. Trying to determine what is "fairest" for all of the thousands of products we buy every year is impossible
We can demand that suppliers respect the labour laws along the whole production chain. That would go a long way. And when that's not enough, we can write stronger labour laws, and enforce them too.