[RobotToaster]

My guess is it's dutifully obeying it, not storing anything from the site and then exiting, without clearing the site from the crawl queue.