They knew full well about other abuse that was done to Jews, opposition and others too. Nazi were pretty bad openly, beating Jews in streets, beating people who shopped in Jewish stores, taking away their possessions, restricting their rights. All that long before they invented death camps.