AWS isn't a utility because utilities sell commodities for which you can switch to a competitor easily, as you're buying the same thing. There are companies selling similar products to AWS but none that sell AWS itself.
Early utilities tend to not be built on standards. See how early electric utilities within a country or even a city couldn't agree on voltage, frequency, phase, or even AC vs DC.
Still, there is a difference of degree. The engineering cost of making one city's electrical grid compatible with the consumers of a different city's grid were much lower (even if one grid was AC and the other DC) than the engineering cost of moving from AWS to a competitor because electricity is not that complicated.
Also, I doubt anyone was referring to electricity as a commodity in the early days before the industry developed standards.