[sradman]

The publishing of the PageRank algorithm was the wrong milestone; my bad. The important milestone was a less well-defined point when search engine users and competitors realized that Google was disruptive.

PageRank was the way forward but the true schlep, in the Paul Graham sense [1], was the brute force continuous hardware scaling required to keep up with the growth of the Internet. DEC needed a technical pivot first and foremost. I've seen no evidence that the AltaVista team understood the technical challenge but I could be easily convinced that bean counters and/or management stifled a promising response; absence of evidence is not evidence of absence.

[1] http://www.paulgraham.com/schlep.html

[dboreham]

My recollection from the time (I didn't work at either place but was in the general area) was that everyone assumed Google were delivering their performance by use of massive hardware resources (I heard 70K machines at the time). So perhaps AV folks just couldn't imagine getting the hardware budget to complete effectively?

It's also worthwhile I think to point out that at that time, it wasn't settled that "search engines" as we know them today (like Google) were _the_ way to use the Internet. There were alternatives such as Yahoo (curated directory), and browser-side catalogs (netscape.com home page) that were much more popular. So it is possible also that AV folks weren't exactly thinking "We have to light the afterburners to go after Google in this immensely important internet search space". They might have been thinking "odd, someone spent $xxxM on hardware to run a search engine, that'll never work out".

[jonathankoren]

WRT to curated directories, in the late 90s when Google came (along with others like Northern Light), it was clear that both directories (Yahoo and Moz in particular) and the established search providers (Yahoo, AltaVista, Metacrawler, etc.) were being overwhelmed.

Yahoo (aka Inktomi) search frequently took multiple pages of to find anything of quality. Curated directories were missing quite a bit. Furthermore, if I remember correctly there was some sort of payola scandal around Moz/OpenDirectory editors.

The market was primed for a better solution, and PageRank provided it.

[Scoundreller]

I always laughed when I saw « Powered by Inktomi », like, why does this company I’ve never heard of want to be associated with this ?

[jonathankoren]

Inktomi was pretty good then. In fact they powered both Yahoo and Microsoft web search at the time. When they were acquired by Yahoo, they formed the basis of Yahoo search, that was well respected in the search research community for years until Yahoo started to implode.

[johbjo]

Early Google used commodity hardware to the point of being comical. Somewhere there is a "historical" display of commodity motherboards spread out on cork trays in a rack. Maximizing density/minimizing cost.

The ability to scale in this way with commodity hardware was new and important.