Hacker News new | ask | show | jobs
by gcmartinelli 5072 days ago
If everyone could implement their own scrapper they would be able to follow robots.txt guidelines/limits, but HN servers might take a big blow it it gets popular.