Hacker News new | ask | show | jobs
by Ajay-p 848 days ago
Does Reddit really block all search engines, or do "all" search engines abide by No Spiders? If it's freely available it's ripe for scraping, no matter what Reddit may say.