Hacker News new | ask | show | jobs
by 2OEH8eoCRo0 1228 days ago
The idea that a robots.txt will save you is laughable.
2 comments
z3c0 1228 days ago
Agreed. At best, you can disallow: / and hope they're polite enough to listen.

I can't seem to find anything on OpenAI's crawler agent, so I'm skeptical they're considering robots.txt at all.

link
2OEH8eoCRo0 1228 days ago
Even if they abide, this is capitalism. Somebody who wants an edge won't. Or OpenAI or Google will get desperate and stop abiding.
link
JohnFen 1228 days ago
True. Robots.txt is already a very weak thing. I disallow all access using robots.txt, but there are many crawlers who ignore it and I have to maintain an overt blocklist for them.
link