[angusgr]

If the URL is a publicly accessible webserver with no robots.txt telling them to stay off it, I wouldn't be surprised if it gets fed to the crawler.

[eli]

Google can even index files listed in robots.txt (it just doesn't crawl them)