[mlyle]

Xerox Grapevine experienced this in 1983. Servers that had free disk space for messages in a congested cluster would announce this, and other nodes would swarm and relocate objects there and you'd get oscillation.

> The moral of the story here is make sure you pick a metric that reacts to the change in request rate as quickly as your request rate changes!

The key things are that you don't react to changes in the metric faster than the metric can move (be appropriately damped), and that you react to the metric "smoothly" (e.g. pick a random server where the odds to get a specific server vary with the loading metric, like you mention).

As others say, it's fundamentally a controls problem... a controls problem where there are many, many actuators and the delay/phase shift is relatively unpredictable. Making things easy for the control system by making reaction smooth and the system overall react slowly (overdamped) is important.