Y
Hacker News
new
|
ask
|
show
|
jobs
by
mikeqq2024
732 days ago
for a long crawling task, if exited/broken for any reason, does it save and restore at the next run?
1 comments
eXpl0it3r
732 days ago
The README says:
> Can resume if process exit, save checkpoint every 250 urls
link
mikeqq2024
732 days ago
nice, better make it as a commandline option with default value. 250 is too many for large files and slow connection.
link
> Can resume if process exit, save checkpoint every 250 urls