Hacker News new | ask | show | jobs
by onreact-com 6177 days ago
Yeah, but do we need Gmail or Facebook crawled? No, both are private. They don't get indexed at all.

The CNN site is quite messed up. Check this:

"2 U.S. Marines killed in Afghanistan" site:cnn.com

You have duplicate content twice + the mobile version does not even contain the article when you click on the search snippet.