Belgium, Norway, Sweden and Austria were also neutral at some points. (Or are still.) And those are just the nations that I can name at the top of my head.
Wikipedia says "Sweden (now EU): has not fought a war since ending its involvement in the Napoleonic Wars in 1814 with a short war with Norway, making it the oldest neutral country in the world."