I used to agree with you until I saw this presentation by Jim Keller, one of the biggest names in chip architecture. It really illustrates how many more options we have to improve performance.
I like that one. Building a chip involves many different manufacturing steps, each of which has room for improvement. I liked how he even mentions that advances in sandpaper allow us to make better chips.