[ido]

Yet somehow the population still declined significantly[0]?

    The Early Middle Ages saw relatively little
    population growth with urbanization well below
    its Roman peak...Estimates of the total population
    of Europe are speculative, but at the time of
    Charlemagne it is thought to have been between 25
    and 30 million

Where as at its peak the Roman Empire alone had a population of 70 million.

[0] https://en.wikipedia.org/wiki/Medieval_demography