You're actievly committing the sink cost fallacy. You need to ignore how much has been spent already and just look at how much it more it will cost from right now to host the Olympics, compared to potential value it would produce.
These comments are old but I figured I should point out the meta: if the japanese government cuts contracts they would just have to spend yet more on subsidies. Japan already has extensive subsidy programs for companies with reduced revenue from Corona. Not only would Japan lose out on any upside, there would not be any savings. Further more: it is Japan: the government is not for want of cash.