|
For the record, I've been crawling the Web since the beginning of DDG (and before the BOSS or the Bing APIs existed). Yes, a one FTE, self-funded search engine, does not have the capacity to spend millions of dollars on crawling (does Google spend billions?). I now focus my crawling efforts where I see they can add significant value, namely on spam removal and zero-click info. Our index actually shows the top result about half the time, which is not from any external API. Given that most people click the top result most of the time, this is not an insignificant addition. We of course use external sources, but I think your comment is too dismissive of the intelligence we've woven on top and through them. I'm not interested in sharing all of DDG's trade secrets just to prove a point, but needless to say, you can simply compare the external APIs to our results across a swath of different types of queries and see the differences. Finally, I find the premise of your comment a bit short-sighted, but I'm actually OK with being dismissed as a toy. In the eyes of the average user, they don't care where results came from at all. They just want the right information faster. So quite frankly, I think it is a reasonable move to use external APIs and focus on things like the top result, UI, etc. More recently, Yahoo and Ask seem to agree. Most people are dismissive of them too, but they've been doing a lot of innovative UI things, which matter a lot to end users. |