[greglindahl]

One thing that is often a surprise to webmasters is that if you have 100+ websites on a single IP, robots.txt specifies a crawl-delay per website and not per IP. So you can end up with 100x the crawling you thought you were specifying in robots.txt.

[johngd]

This is very true. Logistically (not technically) this wasn't an option - read: business decision. Before the days where everything had an API, and things were done with spreadsheets, and billing codes, and product codes, and typos... managing DNS for a large number of domains was not fun and rather expensive. Want to move a bunch of domains to a new IP address? $$$+Time + more time and money. We weren't able to trust our domain broker and had to double check EVERYTHING they did.

To you point, if you are a large provider, especially one that passively and actively sends a lot of money towards the search engines, there are some additional options at your disposal. We (the business units) had contacts with adsense etc, which would come in handy.