[fendmark]

Interesting

Yeah, Facebook's robots.txt file white-lists specific search engines and offer an option to get whitelisted ,then there is a wildcard disallow for everyone else.

Of course nefarious scrapers can ignore the robots.txt file or even spoof google or bingbot, but it least it sets a precedent and a policy that they can take further action on if needed.