[visarga]

> Modern BERT with the extended context has solved natural language web search. I mean it as no exaggeration that _everything_ google does for search is now obsolete.

The web UI for people using search may be obsolete, but search is hot, all AIs need it, both web and local. It's because models don't have recent information in them and are unable to reliably quote from memory.

[nroets]

And models often makes reasoning errors. Many users will want to check that the sources substantiate the conclusion.

[fennecfoxy]

As they should do even for a Google search.

I see search engines as a dripfeed from a firehose, not some magical thing that's going to get me the 100% correct 100% accurate result.

Humans are the most prolific liars; I could never trust search results anyway since Google may find something that looks right but the author may be heavily biased, uninformed and all manner of other things anyways.

[vidarh]

The point is that the secret sauce in Google's search was better retrieval, and the assertion above is that the advantage there is gone. While crawling the web isn't a piece of cake, it's a much smaller moat than retrieval quality was.

[pixl97]

Eh, I don't really see that.

Crawling the web has a huge moat because a huge number of sites have blocked 'abusive' crawlers except Google and possibly Bing.

For example just try to crawl sites like Reddit and see how long before you're blocked and get a "please pay us for our data" message.

[literalAardvark]

My experience running a few hundred very successful shops (hundreds of thousands of orders per month) is that there's no need for quotes around 'abusive'.

95% of our load is from crawlers, so we have to pick who to serve.

If they want our data all they need to do is offer a way for us to send it, we're happy to increase exposure and shopping aggregation site updates are our second highest priority task after price and availability updates.

[vidarh]

It may be tricky, but it's a piece of cake compared to doing good retrieval.