Another thing to look at is using nginx as a web server. It's a lot more lightweight than Apache. Paired with a solid SSD drive and varnish caching, and you can get some really great performance out of WordPress.
I've seen Varnish choke on mainline code: the Recent Comments widget, which can invalidate an entire site whenever a comment is posted anywhere. I have one site which caused both WPEngine and Page.ly to lock up on just 30k views/day because of that.
What I do with my VPS:
1. WP Supercache is configured to write .html.gz copies to disk.
2. Nginx is configured to check if such a file exists before referring anything to PHP. If it finds the .html.gz, it will serve it directly from disk. Ordinary file caching gives it the advantages Varnish promises.
3. WP Supercache appears to correctly handle invalidation when a new comment is posted.
What I do with my VPS:
1. WP Supercache is configured to write .html.gz copies to disk.
2. Nginx is configured to check if such a file exists before referring anything to PHP. If it finds the .html.gz, it will serve it directly from disk. Ordinary file caching gives it the advantages Varnish promises.
3. WP Supercache appears to correctly handle invalidation when a new comment is posted.