Counterexample: the 1984 Summer Olympics in Los Angeles.
Los Angeles had the advantage that they didn't have to build much for the Games. For venues for the events they have enough large arenas and stadiums for the major pro sports teams there and enough universities with large stadiums to cover all the high attendance events.
For the lower attendance sports they had plenty of smaller commercial venues and college facilities.
For housing they used student housing at UCLA and USC. Combined USC and UCLA have around 50k undergraduates with around 40% living on-campus with much of that vacant over the summer.
They ended up only needing to build a couple new venues--one velodrome and one swimming facility, if I recall correctly.
That was then - this is now. I doubt LA would be able to afford the Olympics, if they had to pay for them now. Entire countries cannot afford to host the Olympics anymore - it's time to shut them down for good or revamp them to be actually affordable for hosts and guests.