Hacker News new | ask | show | jobs
by tecleandor 462 days ago
Nice. How do we force them to respect robots.txt?
1 comments
szczepano 462 days ago
they respect robots.txt at least major ones like meta, claude, google, openai, based on my infra observations robots.txt is enough in 90%, 10% is just banning ip ranges for couple of days but those are no AI companies
link