Hacker News new | ask | show | jobs
by hachiya 6251 days ago
Is this truly crawling the sites, or querying each site's search service?

e.g. for Twitter, are you crawling all the Twitter users' URLs (seems difficult to do - how do you find new accounts' URLs?), or are you fetching results from search.twitter.com ?

3 comments

We take the raw data streams (e.g. a data mining feed from Twitter) and process them, ourselves.
Yeah. I'm curious if they developed something similar to Twitter's search functionality or they are using Twitter's Trends feed.

In its current form, most of the results are from Twitter which doesn't really differentiate it much from Twitter Search.

I don't think crawling and querying are mutually exclusive ideas.