[yummyfajitas]

I imagine Koreans didn't move to Virginia after the war.

I thought "colored" == "not white" was common knowledge, but if you want a specific definition, take Virginia:

For the purpose of this act, the term “white person” shall apply only to the person who has no trace whatsoever of any blood other than Caucasian; but persons who have one-sixteenth or less of the blood of the American Indian and have no other non-Caucasic blood shall be deemed to be white persons.

https://en.wikipedia.org/wiki/List_of_Jim_Crow_law_examples_...

Colored explicitly includes "Negro, Mongolian, American Indian, Asiatic Indian, Malay".

As for whether I believe human populations can rapidly rebound, the answer is yes. I believe this because so many human populations have actually done this. I've cited many examples and they in addition to suffering poverty and oppression far worse than any in the US, they also satisfied your criteria of having Jim Crow laws applied to them in the unlikely event that they were permitted to enter the US.