The problem is that the routers that have to hold the routing table can only handle a limited number of routes. There is a good article from APNIC about the topic.
And besides, there is no need to keep the whole routing table in RAM. Instead all that's necessary is a single integer per route representing which port packets to each route needs to be sent down. So even for a large router with 64 ports, the whole routing table fits inside 1 megabyte.
Sure they can create routers with more space for the routing table but you still have to replace all the old ones which isn't cheap.
A single port of a router at an internet exchange can reach more than 1000 different routers. A router has to decide to which IP it should forward a packet not just the port.
And besides, there is no need to keep the whole routing table in RAM. Instead all that's necessary is a single integer per route representing which port packets to each route needs to be sent down. So even for a large router with 64 ports, the whole routing table fits inside 1 megabyte.