[logicslave]

Funny thing I was thinking about recently...why not reverse engineer it? Run the top million most common queries, or maybe top billion, snapshot the top 100 results, use that to train your model. With cheap enough compute, can google be reverse engineered?

[notshift]

You'd have to feed your model a copy of the entire internet, and if you can do that, you've already done the hard part of creating a Google clone imo.

In general, if folks want to know how Google works, just do some reading on grey hat / black hat SEO. There is an entire (somewhat) underground industry of people that have ranking in Google down to a science - put exactly this on your page, set up exactly these linking domains with exactly this type of content, satisfying all of these metrics, etc. I honestly think the reason competing search engines are so much worse is just because none of them have tried very hard, or maybe because they just lack funding.

AFAIK, the algorithm is still the core of what it always has been (getting PR links to your page) but Google has just added a bunch of layers on top of that which basically check for things to disqualify you completely or make minor adjustments to your position in the rankings.