[iigs]

Approach 1)

A long time ago I worked (in a very small testing capacity) at a very large video/streaming media serving company.

As I recall, the scripted procedure was to tag a release in CVS, and a script would pull that down into a new directory and swing the symlink.

This site had a very mature and infrequently changing billing system code path, which I believe was not modified in this process. If this doesn't describe your workload you would probably have to change this to support concurrent execution of multiple versions for people who have a session established

Approach 2)

Using any mainstream hardware load balancer (or presumably a similarly featureful free/open software LB), configure it to point at N+1 machines in a cluster. Administratively remove machines from the new session pool one at a time (virtual machines can make this flexible and easy to roll back). Once the established sessions have expired or been forced out, upgrade the software and roll them back in.

One neat aspect of this approach is if you have an "oh shit" hockey stick scaling issue, you can watch it happen on one machine before deploying it to every machine in your cluster. Also good for A/B testing, as mentioned in recent articles here.