[metronius]

If you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.

[psykovsky]

You say that like it doesn't happen all the time. googlebot included.