Hacker News new | ask | show | jobs
by dexen 5445 days ago
tl;dr: Yes they are indexed separately, and yes it's up to site owner to decide which content may be indexed for Google News -- via proper META tags and robots.txt.

There is an important difference in UI between Google Web Search and Google News: the Web Search presents a very short snippet of text and link, while Google News presents a bit more of the news content. Effectively, Web Search directs viewers straight to the original website, while News lets them just browse Google News, sending less traffic to the original website. [1]

Which means Google News may provide less ad revenue to original website -- probably the crux of the problem here.

To handle that, Google provides two different bots: one with user agent `Googlebot', the other with `Googlebot-News'. If a website wants its content only in Web Search, and not in News, it's supposed to return proper /robots.txt -- and that does the trick. Way quicker and cheaper than any lawsuit!

http://googlewebmastercentral.blogspot.com/2009/12/new-user-...

----

[1] but hey, less is more! whoever makes it to the original site via Google News has clearly show elevated interest in the news piece, and thus should be worth more to ad providers.