Hacker News new | ask | show | jobs
by inigyou 17 days ago
Scrapers generally aren't looking for random websites to scrape. They have a specific URL in mind. Only if the goal was DDoS would they not care which URL was accessed.
1 comments

They also don't care which URL was accessed when the goal is scraping as much text as possible for AI training.