Lettuce be honest. The end of slavery in the South was immediately replaced by slavery-like exploitation of second-class workers without enforceable rights. When a massive nationwide movement for equal civil rights finally made that non-viable a hundred years later, the farmers just did a global search-and-replace in their business plans from "black" to "foreign" and continued as usual.
Having zero abusable second-class workers would be more destructive. 150 years of dragging ass on modernizing their farming operations will come back and bite them all at once. But they probably won't learn their lesson and just use robots. They will beg and squeal for prisoner labor first. And then the cops will be ginning up bogus traffic offenses right before harvest time, so the people who can't pay the fines will have to pick vegetables instead.