Hacker News new | ask | show | jobs
by mymacmachine 977 days ago
I'd imagine a different dystopian nightmare.

Google don't care if an individual site blocks them (in my experience they are one of the better crawlers at obeying robots.txt) because they know - given their search monopoly - that the vast majority of the internet would never dare to cut them out. WEI is a way to cement their current position further without needing to play dodgy games with special Google certs.

If they put out WEI and start encouraging sites to use it for better SEO, the majority of the internet will do it (and wont put in the same effort to support other crawlers because those search pages are only < 1% of traffic according to their Google Analytics)

At that point, when change (legal, technical, social, whatever) comes to try and break apart their hold on search - there is yet another technical constraint that works in Google's favour. It won't be Google's "fault" that websites give them privileged crawl access and greater reach, but we'll all be stuck with it.