It's not since it can be a great source of pride for people to see their nation represented by their best athletes. It's a display of the best humanity has to offer in the areas of physical endeavor, discipline, and endurance. Reducing everything's value to a function of its consumption of money and resources is a depressing and inhuman way of viewing the world and thankfully most people don't think this way. We're not robots.
It can be done conducting at one place instead of building and then leaving behind infra. See what happend to what was built last time in Brazil and guess how many Brazils are happy to hear what you said