The Irish & Southern Europeans were looked upon as lesser races at the time. It was only later that "white vs black" became the socially accepted racial classification.
I remember reading, quite a while ago, some 19th or 18th Century piece of fiction in which a Caucasian lad with black hair was referred to as a "little black boy".