Hacker News new | ask | show | jobs
by ithkuil 5695 days ago
Well, you can try it with:

wget -O- http://www.boutiques.com/robots.txt -U "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

and you get the same error.

I tried a couple of other bot user agent strings from http://www.useragentstring.com/, same result