Hacker News new | ask | show | jobs
by mateozaratefw 701 days ago
Tik Tok crawler fucked us up by taking a product-name (e-commerce) and inserting it into the search bar recursively with the results page. Respect the game, but not respecting the robots.txt crawl delay is awful.