[etep]

If it was easy to build high performance caches with high associativity, we would certainly see higher associativity. Ideally, you want a fully associative cache, but it's too expensive. In CPU caches, once a set is selected, all N associative ways are compared simultaneously. So growing associativity costs area and power for extra comparators. This growth could cause timing issues, i.e. add latency to the memory access or cause CPU freq. to be lowered.