[briandear]

I don't know why this was downvoted, but this was obviously the case. CL had no interest in these 3d party sites. It's their data and their terms of use, so there really aren't any debatable points. Nobody has a right to CL data unless they explicitly provide that right. If I were CL, I wouldn't have give Padmapper an API either; their arrogance was astounding. They had no respect for the CL terms of use. They built an entire company around a knowing violation of their data source's terms of use. The Terms said, "you can do X," Padmapper did "X" then complained when CL said no. It doesn't matter what X was, Padmapper just knowingly did it anyway. They had the intent to violate the Terms of Use.

[sangnoir]

It's amazing how wholesale scraping of data was being celebrated, because the scrapers "were making the internet better".

This is the first topic I've seen were the HN's usual Libertarian bias disappears and no-one challenges the underlying notion that CL is seen as part of the Commons. Channelling Dagny Taggart, I'd say 3Tap & PadMapper were a bunch of moochers.

[rev_bird]

You're making it sound like the scraping was being done to replace the functionality of CL, which, yeah, would be pretty transparently shitty to do. But they weren't doing that, especially PadMapper: They were indexing the content to make it more accessible, an action that's been taken probably trillions of times and is pretty much the main reason most of the internet is even usuable today. It's like accusing Google of plagiarizing your website because they linked to it.

[mark-r]

That's a good point - does craigslist have a robots.txt to prevent Google from crawling it? If not, isn't Google guilty of the very same thing, by aggregating the information via search results?

[makomk]

Craigslist doesn't prevent Google from crawling them. Not only that, Craigslist also sued at least one company for scraping Google results in order to index Craigslist postings.

[rev_bird]

> User-agent: * > Disallow: /reply > Disallow: /fb/ > Disallow: /suggest > Disallow: /flag > Disallow: /mf > Disallow: /eaf

Nothing blocking listings... OR PadMapper...

[mikerichards]

This is the first topic I've seen were the HN's usual Libertarian bias disappears

Not even close.