Hacker News new | ask | show | jobs
by js4ever 1836 days ago
you should rate limit this, or it's a DDOS as a service ...
1 comments
owor 1836 days ago
We use an auto-throttling algorithm for the crawler to ensure we are a 'polite' crawler!

We also send a unique user-agent with every request, so any host can easily block us if required - luckily, this has never happened so far.

link