On the contrary, having driven plenty of cars with and without AWD, adding AWD is a massive boon to handling for me, even outside of "rough" conditions like mud/ice/snow/gravel.
The issue is the height, weight, etc, not really the AWD.
Especially since most vehicles’ AWD systems (especially if you’re talking crossovers) are really just “2WD but we can briefly engage the other two wheels if you’re trying to get moving on a patch of ice”. (One axle is not being driven except when the other is slipping, and in many cases there are speed limits above which this won’t engage. Many of these systems rely on clutch packs that, if they were engaged continuously, would overheat.)
Especially since most vehicles’ AWD systems (especially if you’re talking crossovers) are really just “2WD but we can briefly engage the other two wheels if you’re trying to get moving on a patch of ice”. (One axle is not being driven except when the other is slipping, and in many cases there are speed limits above which this won’t engage. Many of these systems rely on clutch packs that, if they were engaged continuously, would overheat.)