The switching faster bit has only been happening very incrementally with the breakdown of Dennard scaling. Thankfully the less energy bit seems to be part of a more fundamental process than Moore's Law[1] and there are nice, clear, theoretical limits on how power efficient a computation can be which we're nowhere near hitting[2].
[1]https://en.wikipedia.org/wiki/Koomey%27s_law
[2]https://en.wikipedia.org/wiki/Landauer%27s_principle