Hacker News new | ask | show | jobs
by alphast0rm 4933 days ago
The tricky thing when doing this is knowing what rate to stop at without getting permanently banned. I built an Android Market crawler two summers ago, and luckily Google only temp bans (from my experience), so that might be an easier project without any risk.
2 comments

Respecting robots.txt is probably the best plan.
Use disposable IPs.