Hacker News new | ask | show | jobs
by greglindahl 3584 days ago
A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.
1 comments
sjwright 3584 days ago
I agree, it's a difficult conundrum. It sucks.
link
greglindahl 3584 days ago
There are worse barriers to entry for a search engine! DMCA take-downs... Right to be forgotten... click history...
link