[jwr]

This really depends on the specific architecture and the application. In some cases, you will want to optimize mostly for size, so that your hotspots fit entirely into I-cache. Modern CPUs spend most of their time waiting for data (or instructions) to become available, so often computations are essentially free.

[gpderetta]

the average IPC over a variety of loads is, IIRC, estimated to be ~1. So no, most modern cpu do not spend most of their time waiting for data.