It's far from a "few idiots". The place is super controversial. A nice place, but super controversial, politically.
Like I say above, why do a few idiots waving flags at Meiji mean that _the entire country_ needs to be "taken down" like the OP says?
I stress that this is "Japan"- the whole of Japan. Not the idiots waving flags at Meiji. Are all Japanese nationalist assholes?