Y
Hacker News
new
|
ask
|
show
|
jobs
by
dzhiurgis
710 days ago
Google sometimes ignore it when it makes sense (ie big bank accidentally adds login page to ignore) or to check for spam activity (in which case google doesn't use their bot user agent)