Yet somehow the population still declined significantly[0]?
The Early Middle Ages saw relatively little
population growth with urbanization well below
its Roman peak...Estimates of the total population
of Europe are speculative, but at the time of
Charlemagne it is thought to have been between 25
and 30 million
Where as at its peak the Roman Empire alone had a population of 70 million.
[0] https://en.wikipedia.org/wiki/Medieval_demography