Hacker News new | ask | show | jobs
by dzhiurgis 710 days ago
Google sometimes ignore it when it makes sense (ie big bank accidentally adds login page to ignore) or to check for spam activity (in which case google doesn't use their bot user agent)