[mdasen]

How do you know if they're malicious if you don't make HTTP requests to them?

One of the things that phishers and others do is use link wrapping and other services to hide malicious links. So, I get something.wordpress.com/something-clean. I then put in an HTML or JS redirect on that page to something malicious. Given that browsers don't warn about HTTP, HTML, or JS redirects, it's an easy way for scammers to get around a list of malicious pages.

These kinds of attacks are very common in the email space.

[gruez]

But in this case, that doesn't help at all because facebook's crawler uses a predictable user agent string. You give a clean result to the facebook crawler and a malicious result to everyone else.

[seanieb]

There are services to frawl for you from miltipke ips and user agents, just for situations like this.

[idoco]

That is a very good point. Security crawlers should probably use a masked user-agent.

[marksomnian]

I'm fairly sure Google's search crawler already uses a masked UA, to detect when pages serve it different content than they do to users.

[allie1]

Not always, it masks UA and IPs when checking for ads content to uncover cloakers, so its within theit codebase to do this. Not sure why they’re not using it here.

[bluntfang]

>How do you know if they're malicious if you don't make HTTP requests to them?

look-alike domains are phishing vector that don't require you to make an http request.