Routing tables need to be limited in size in order for hardware to decide where to route packets fast enough. In the past hardware was more cpu and memory bound than the present.
But also at present, dealing with millions of routes at the packet-switching level gets difficult. Especially because the speed of links has increased so much.