The switching faster bit has only been happening very incrementally with the breakdown of Dennard scaling. Thankfully the less energy bit seems to be part of a more fundamental process than Moore's Law[1] and there are nice, clear, theoretical limits on how power efficient a computation can be which we're nowhere near hitting[2].