Helpful quotes on optimization fall in to two categories, first - the fallacy of optimization in general.

Optimization: Prototype before polishing. Get it working before you optimize it. - Eric S. Raymond, The Art of Unix Programming (2003)

Optimization considered harmful: In particular, optimization introduces complexity, and as well as introducing tighter coupling between components and layers. - RFC3439

Spell create with an 'e'. - Ken Thompson (referring to design regrets on the UNIX creat(2) system call and the fallacy of premature optimization)

The First Rule of Program Optimization: Don't do it. - Michael Jackson

The Second Rule of Program Optimization (for experts only): Don't do it yet. - Michael Jackson

Second, the established method.

Before optimizing, using a profiler. - Mike Morton

... from my fortune clone @ https://github.com/globalcitizen/taoup

Couldn't the minimum of 2 iid rvs drawn from a pareto distribution be calculated analytically?

The article while interesting doesn't even attempt to talk about identifying _WHY_ he is experiencing latency spikes.

It seems to me that the first step is to instrument the heck out of your application. That way you know _WHY_ it sometimes takes longer to respond than at other times. I understand this is harder in a GC'ed/JIT'ed language, but its not really an excuse.

Good instrumentation is helpful for far more than tracking and isolating problems in ones software stack. The project I work on has pretty extensive request level instrumentation which tracks each request at multiple points as they are processed. More than once the first signs that a piece of hardware is failing is the disk or network IO's start to have unusual latency patterns. Often software performance regressions show up as highly variable database requests, etc..

This is interesting. Of course you now have to synchronize state between the servers, and it will only work if a significant fraction of the higher-latency events are independent between servers.