Hacker News new | ask | show | jobs
Google boutiques.com has no robots.txt (boutiques.com)
6 points by techvibe 5695 days ago
5 comments

Well, you can try it with:

wget -O- http://www.boutiques.com/robots.txt -U "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

and you get the same error.

I tried a couple of other bot user agent strings from http://www.useragentstring.com/, same result

Why would it?
they have NOINDEX, NOFOLLOW robots META tags that tell search engine crawlers to go away.

Maybe it's their way of playing nice with all of the merchants they are listing.

Could it be user agent filtering?
So?...