Hacker News new | ask | show | jobs
by Alifatisk 905 days ago
What's the consequence of installing the crawler to FF? Can the ISP / Cloudflare / any other party start blacklisting you?
2 comments

Well, the internet could know the balance of your bank account at that boutique credit union, or which sex toys you’ve been looking at buying.

We do web hosting for businesses. We are careful to identify bots because we don’t want personalized recommendations in search engines, and don’t want PII showing up. Not that we put a lot of that in the site, but think about a Home Depot or a Best Buy situation. If you’re a human near the Minneapolis store, you want availability for that store, not the Detroit one. But corporate wants a bot to know that this microwave is something you in theory carry, but might have to ship from a warehouse. If Home Depot wants that, and our customers also want that, then probably a lot of businesses feel the same.

But the extension is not crawling my stuff, it takes batches of urls from the host server. How is the internet going to get my bank account?
> Can the ISP / Cloudflare / any other party start blacklisting you?

Yes, wouldn't surprise me if you were to get blocked from tons of sites and flagged as a bot on Cloudflare and other WAFs.

Yeah getting blocked from Cloudflares network is not something I want to experience, it's a hell.