Hacker News new | ask | show | jobs
by gregwebs 3 days ago
This seems like it has potential to create a lot of load on a site- are there settings to set how fast it clones or avoid images/videos? Is there a way to only get a subset of a website?
2 comments
tamnd 3 days ago
Could you help create a new issue for that? I will do it later. It is already 1:00 AM my time, but I am happy that anyone is interested in it. : )
link
ares623 3 days ago
Just pretend you're an AI crawler problem solved
link