A big example is Gov.UK which is mostly mostly open source as well as a few other big UK gov services like police.gov [1]
A large part of the NHS was moved off of a proprietary oracle system just a few months ago [2]
[1] http://github.com/alphagov [2] http://www.theregister.co.uk/2013/10/10/nhs_drops_oracle_for...