Hacker News new | ask | show | jobs
by ThomW 384 days ago
It ticks me off that bots no longer respect robots.txt files at all. The authors of these things are complete assholes. If you’re one of them, gfy.
2 comments

You might as well say gfy to anyone using chat bots, search engines and price comparison tools. They're the one's who financially incentivize the scrapers.
That doesn't logic.

Giving someone a "financial incentive" to do something (by gasp using a search engine, or comparing prices) does not make that thing ethical or cool to do in and of itself.

I wonder where you ever got the idea that it does.

Because there's a market to serve, that you refuse to serve, so the stop gap solution is for third parties to acquire the liabilities and risks for compensation.
Financial incentive ≠ moral justification.

You're shifting the frame, to one where morality doesn't come into it. You're asserting some kind of market inevitability, which is probably the same sort of rationalization arms and people traffickers use to sleep at night.

Honey potting the robots file is handy for the ones who don’t just ignore it but go looking for trouble.