Hacker News new | ask | show | jobs
by mccorrinall 1289 days ago
If they don’t want to be spidered maybe just use robots.txt

They can’t have both: exposure AND content only available on their platform
1 comments
chownie 1288 days ago
Can you expand on that? Should Google have rights to use any of your content if you allow spidering? Including for commercial gain?
link
mccorrinall 1285 days ago
Yes and yes. It’s okay to crawl a site and show the meta tags like short description, title, author etc. Why should it be illegal?
link