Sexual reproduction uncorrelates the selective pressures over each gene, and each small set of them (with larger sets being possible the larger a population the species has).
The gene is the unit of selection is perfectly right. It's missing a finer point in that groups of genes are unities of selection too, but it's correct.
This is a common misconception. The collection is the unit of replication, but the gene is nonetheless the unit of selection. This argument was laid out in excruciating detail by Dawkins, but the TL;DR is that reproductive fitness can only ever be measured relative to some environment, and all the other genes in a replicative unit are (part of) the environment for a gene and its alleles.