Well then you're omitting the group that transitioned from productive to dependent from old age. If the other numbers are any clue, then it is over a million.
Right and with fewer newborns replacing the retirees (who exit the labor productive market) fewer workers will have to support a larger older pop. That's the biggest issue --who will provide the tax revenue to support retirees, secondarily the economy becomes less vibrant with fewer minds entering the workforce.