There are about 200 generations between now and then. Your ancestor set needs to only grow by about 10% each generation to include everyone who was alive back then and therefore also them. In a perfectly mixed population your ancestor set grows 100% each generation until it includes everyone. Now humans aren't a perfectly mixed population but you only need a few migrants each century between different societies to reach 10% because in each society it's much closer to 100% (until it includes ~everyone in that society).