By "racist background" they mean "racist association". Perhaps it didn't start that way but due to history it can easily be twisted to be seen as offensive.
I can't tattoo a swastika on my arm, despite its origins.
anything to back this up? my searches mostly come up with either greeks physically using black ballots to vote to excommunicate people, or putting names in a black book in 1600s, which was usually reserved for enemies and most likely was not racially motivated.
Rosstex posted a good explain below.