Hacker News new | ask | show | jobs
by jd20 752 days ago
I didn't see it mentioned, but why not just use robots.txt? Does Bytespider ignore it?
2 comments
hajimuz 751 days ago
Yes. It ignores robots.txt.
link
chptung 752 days ago
Not super sure so this felt like a faster way to plug it ASAP.
link