Hacker News new | ask | show | jobs
by imilk 340 days ago
Having perplexity or other AI bots go haywire and sending 10s of thousands of requests per minute to your website (despite you having a robots.txt blocking them) is a giant pain in the ass. Not only does your server costs go up, but your analytics and attribution reports start to look messed up because of all the bot traffic.
1 comments

Yeah obviously if they're being abusive that's a problem but that's not what the article seems to be talking about.
Well besides them being abusive, the other issue is that AI overviews and answer boxes cannibalize traffic to websites, leading to less conversions for the original content producers. This is pretty well established across industries at this point:

https://ahrefs.com/blog/ai-overviews-reduce-clicks/

That's how people want to browse the web. If you block it you won't even get links from those. That's like blocking the search crawler.
...that's literally the entire point of this article. People don't want their websites being de-listed from the monopoly that controls organic traffic. At the same time they would like some control over stopping companies (in this case, the same company that controls the organic search monopoly) from scraping and repurposing their content so their the traffic to their website doesn't decrease.

Why is it such an issue that publishers and website owners want to maintain the traffic to their website so that they can continue operating as usual? Or should we all just accept every Google decision, even when those decisions result in more engagement on google.com, but 20-35% decreases in traffic to the original websites?

Also I'm going to need a citation that the vast majority of people want and get value out of AI overviews. Because that is certainly not the case from my experience.

Google absolutely is not the only company doing this and if they didn't do it I'd feed the results into my local models to get the same thing.

This isn't a "google decision" people are changing the way they use the web.

Google has clearly decided to keep users on their platform longer, hoping that this will lead to more ad clicks. There is a clear reason why AI overviews very seldomly link to outside websites, and why website links are much more hidden on Google Maps/Business Profiles. More time spent on the Google platform means more likely that someone will eventually click an ad.

Also - I noticed a pretty huge outcry when AI overviews were introduced to search. Can you show me all the people who enjoy the experience of using them more than not?