tl;dr: Yes they are indexed separately, and yes it's up to site owner to decide which content may be indexed for Google News -- via proper META tags and robots.txt.
There is an important difference in UI between Google Web Search and Google News: the Web Search presents a very short snippet of text and link, while Google News presents a bit more of the news content. Effectively, Web Search directs viewers straight to the original website, while News lets them just browse Google News, sending less traffic to the original website. [1]
Which means Google News may provide less ad revenue to original website -- probably the crux of the problem here.
To handle that, Google provides two different bots: one with user agent `Googlebot', the other with `Googlebot-News'. If a website wants its content only in Web Search, and not in News, it's supposed to return proper /robots.txt -- and that does the trick. Way quicker and cheaper than any lawsuit!
[1] but hey, less is more! whoever makes it to the original site via Google News has clearly show elevated interest in the news piece, and thus should be worth more to ad providers.
The longer answer is that the newspapers are struggling financially, Google is not struggling, when snippets of content appear on the Google news page the newspapers believe they are 'improving' the Google brand without being compensated. They sued to have either Google pay them, or remove them from the News site.
What they fail to realize (and Google no doubt knows) is that if you aren't listed in Google you're not going to get nearly as much traffic (hence 'profit'). So Google apparently took a literal interpretation of the order and struck them from the books as it were, complete removal from the index.
So the newspapers have to make the following evaluation, 'what is it worth to us to be in Google's search results?' And is that value more or less than what we think they owe us for having our results on their news page?
Google clearly understands that its value for them to be in the search results and I believe they would say off the record that the value of being in the results is worth more than the ad revenue they get from the occasional story that makes it to the front page of Google news.
Sort of extortion, sort of natural consequences. Humorous though.
There is an important difference in UI between Google Web Search and Google News: the Web Search presents a very short snippet of text and link, while Google News presents a bit more of the news content. Effectively, Web Search directs viewers straight to the original website, while News lets them just browse Google News, sending less traffic to the original website. [1]
Which means Google News may provide less ad revenue to original website -- probably the crux of the problem here.
To handle that, Google provides two different bots: one with user agent `Googlebot', the other with `Googlebot-News'. If a website wants its content only in Web Search, and not in News, it's supposed to return proper /robots.txt -- and that does the trick. Way quicker and cheaper than any lawsuit!
http://googlewebmastercentral.blogspot.com/2009/12/new-user-...
----
[1] but hey, less is more! whoever makes it to the original site via Google News has clearly show elevated interest in the news piece, and thus should be worth more to ad providers.