Hacker News new | ask | show | jobs
by janmo 78 days ago
His robots.txt explicitly allows bots including LLM bots to scrape his site
2 comments
jefposk 78 days ago
The LLM scraper bots ignore robots.txt
link
xigoi 77 days ago
Permission to scrape is not permission to DDoS.
link