[dboreham]

My recollection from the time (I didn't work at either place but was in the general area) was that everyone assumed Google were delivering their performance by use of massive hardware resources (I heard 70K machines at the time). So perhaps AV folks just couldn't imagine getting the hardware budget to complete effectively?

It's also worthwhile I think to point out that at that time, it wasn't settled that "search engines" as we know them today (like Google) were _the_ way to use the Internet. There were alternatives such as Yahoo (curated directory), and browser-side catalogs (netscape.com home page) that were much more popular. So it is possible also that AV folks weren't exactly thinking "We have to light the afterburners to go after Google in this immensely important internet search space". They might have been thinking "odd, someone spent $xxxM on hardware to run a search engine, that'll never work out".

[jonathankoren]

WRT to curated directories, in the late 90s when Google came (along with others like Northern Light), it was clear that both directories (Yahoo and Moz in particular) and the established search providers (Yahoo, AltaVista, Metacrawler, etc.) were being overwhelmed.

Yahoo (aka Inktomi) search frequently took multiple pages of to find anything of quality. Curated directories were missing quite a bit. Furthermore, if I remember correctly there was some sort of payola scandal around Moz/OpenDirectory editors.

The market was primed for a better solution, and PageRank provided it.

[Scoundreller]

I always laughed when I saw « Powered by Inktomi », like, why does this company I’ve never heard of want to be associated with this ?

[jonathankoren]

Inktomi was pretty good then. In fact they powered both Yahoo and Microsoft web search at the time. When they were acquired by Yahoo, they formed the basis of Yahoo search, that was well respected in the search research community for years until Yahoo started to implode.