[dclara]

Thank you for your explanation. Now I understand better. But a distributed system is usually defined in the second meaning of what you said inside the search engine, instead of across different websites.

I'm very interested in this topic because I've proposed my solution for how to improve search engines, not from algorithm point of view, but from systematic point of view. And making it fully distributed is the key.

While you mentioned about the 2-level search and "receiving their algorithms and data ...", I don't think it's very feasible. Do you agree? So vertical distributed architecture across various industries is not a feasible solution. But we can do a horizontal distributed architecture which will collect data from geographic locations. In each location, there will be many different verticals. It's matter of time if Google cannot find a better solution, search engine will be improved in certain way.

[wslh]

While you mentioned about the 2-level search and "receiving their algorithms and data ...", I don't think it's very feasible. Do you agree?

Why not? I don't get it.

[dclara]

Because to my understanding, 2-tiered search means that the routing search engine scrapes data from the second level search engines and return it to the users. The second level search engines, e.g., Stack Overflow, are usually running by separate entities from the routing search engine, say DDG. If DDG does not own all the second level search engines, how can they get the local indexes and ranking algorithms from them? And even if DDG does get it, it's no longer decentralized any more. So what's the difference from Google?

[wslh]

The difference with Google is this:

1) Google quality of indexing doesn't have any competition yet.

2) They can calculate a page rank across different domains

3) No single entity can make the same efforts or is so smart to build a similar thing

If you follow the 2-tier route:

1) Each entity takes responsability to optimize the quality of search locally.

2) They know their own domain or they can learn how to optimize their page rank at a local level instead of a global level

So, at the end you have distributed the work of local optimization across different intelligent entities. For example, when you look at the Linux kernel or other open source projects you can count million of man hours that are difficult to have if you run a single entity.

[dclara]

Yes, I agree with you on using a 2-tier search which will increase the relevancy to optimize the quality of search. And Google's search quality is not unbeatable.

I also agree with you by using distributed sites to optimize the results locally. Actually what I proposed is to make the distributed search from both geographic location and vertical market point of view, as opposed of dedicated sites from you. But they are complementary. The dedicated sites definitely will provide better and more relevant results than a global search engine if Google search was not limited to a particular site.

However, the only thing I don't agree with you is when you said it does not have to be distributed though, the search router can integrate the algorithms from the dedicated sites. Then I think it's not quite feasible since it's not possible for Stack Overflow or Wikipedia to share their algorithms with DDG.

Let me know if I misunderstood you. If you'd like to take if offline, I'll be happy to discuss with you via email. See my profile.

[wslh]

Why it's not feasible to share Stack Overflow algorithms with DDG? they just give them the application to run it locally.

[dclara]

What? DDG does not own Stack Overflow but it can run Stack Overflow's algorithm locally? Do you mean that Stack Overflow has it's search algorithm open sourced? I'm sorry, I don't quite get it.

Usually, the router search engine queries data from the second tier websites to get high quality results without having other websites' algorithms. Also there is another problem: how do you know which websites to go for given an arbitrary keywords? For example, when user searches for "cookie" on your search engine, where do you send the query to? How do you know if they are looking for food cookie or browser cookie?