I agree. Vegans are usually 'all or nothing' when it comes to animal welfare. However, if you could increase the quality of life of farm animals from miserable to decent for the majority of their lifespans, that would surely eliminate the majority of animal suffering overall. I don't get why someone who claims to be concerned about animal welfare wouldn't promote such efforts.
I find it a bit silly because "ethical" is a vague term that can be interpreted in many different ways. For instance, some people will still not think that the dairy's practices are "ethical", if the end result is the same, i.e. slaughtering the animals.