[hinkley]

In late 2000 at the tail end of the bubble bursting, there was a search engine company trying to build on a platform of push notifications instead of crawling.

I know that at my current company the plurality if traffic comes from crawlers. We don’t want to throttle them because that’s biting the hand that feeds but it sure sucks.

I wonder often, how many crawlers you need before it’s cheaper for a website to volunteer up when pages change or new ones arrive.

[th0ma5]

Maybe Pointcast? https://en.wikipedia.org/wiki/PointCast_(dotcom)

[greglindahl]

There are N existing standards for websites to notify crawlers about new content. Few websites use them, and the ones that do are often buggy. A few are even malicious.

I'm looking forward to the N+1th standard.