[gruez]

But in this case, that doesn't help at all because facebook's crawler uses a predictable user agent string. You give a clean result to the facebook crawler and a malicious result to everyone else.

[seanieb]

There are services to frawl for you from miltipke ips and user agents, just for situations like this.

[idoco]

That is a very good point. Security crawlers should probably use a masked user-agent.

[marksomnian]

I'm fairly sure Google's search crawler already uses a masked UA, to detect when pages serve it different content than they do to users.

[allie1]

Not always, it masks UA and IPs when checking for ads content to uncover cloakers, so its within theit codebase to do this. Not sure why they’re not using it here.