The genetic differences could be down to socio-economic factors though, such as some of your ancestors having been slaves and restricted in how they could choose a partner.
Not really. It works on the timescale that specific causes are affecting the genepool. Europeans with below average immune systems were removed from the gene pool virtually overnight by the black death. Native Americans famously never had the 'benefit' of this event.
Similarly, there was an enormous selective pressure on Black slaves in the Americas, so it's certainly possible some effects took only a generation to be reflected in the gene pool.
However, when it comes to intelligence, no strong evidence exists saying it is genetically predetermined. Culture and rolemodel of parents are far greater determinants, implying selective pressure is unlikely to affect the general intelligence of a subgroup.