It seems to be used both with and without.
In the main table with the maps it's not directly after, there's a comma. The ", England" is specifying the country, same as where it says "Yorkshire and the Humber, England", "East Midlands, England", etc. There's no comma part for "East of England", because there it is part of the name. In the demographics table it is directly followed by "England".
A more official source puts "England" in brackets, as if it's not part of the name but just to disambiguate: https://www.ons.gov.uk/methodology/geography/ukgeographies/e...