It doesn’t mean the population fell that low. It means some sub population that size happened to be the ancestors of today’s humans. Nothing is proven about why the other sub populations aren’t represented today or when they died out. It could have been gradual, after our common ancestor group had successfully outcompeted the other groups.