Hacker News new | ask | show | jobs
by warrenm 1258 days ago
only if the scraper/crawler actually pays attention to robots.txt
1 comments
dutchbrit 1258 days ago
Which they ultimately should do when crawling a site. OpenAI does according to ChatGPT.
link
warrenm 1258 days ago
as a coworker of mine liked to say, "'should' is a funny word"
link