[pkandathil]

Yeah. I think that is the challenge. A good way to get around the AJAX problem is to see if a site has an RSS feed and use that to extract content. I wish sites had a url for bots built in so you didnt have to do all this fancy stuff to extract the content.

[buss]

Many of the big sites will feed you non-ajax content if you're the googlebot.