[wongarsu]

New reddit is more annoying (to me as human, but also to scrapers) in two very important ways:

- old reddit shows more or less the raw comment tree, sorted, like HN. New reddit does aggressive folding. You get a couple comments for each branch and have to click "show more" to see more

- new reddit forces you to log in to view anything tagged NSFW, including entire subreddits. Old reddit does not enforce this, along with not enforcing some other restrictions reddit has introduced over time

Of course none of this stops a scraper. And there is still the API. But these things make it easier to detect scrapers because they have to make a lot more requests and have to handle user sessions that make them easier to track

[tlonny]

As I understand it, anything publicly viewable is free to be scraped without issue.

If you scrape stuff behind a sign-up, you potentially open yourself up to being sued for breach of ToS.

Now I'm sure the Chinese labs don't give a rats arse about that threat, but I imagine it stops the likes of Google, OAI, Anthropic from scraping and instead forces them to purchase the data.

[benjiro29]

Not how it works ... Anything public viewable is still subject to a TOS. And the whole copyright or whatever still applies.

If your argument was valid, we can scrape news websites and show their content freely. No ... You get sued and lose the case.

See Google News that got their behinds in court and lost. Short summaries are allowed / transformative, but you can not just take content (even without a login). Not without getting into civil court if somebody wants to press the matter.

Now, if you scrap and never make that data public or transform it (LLMs). Then it becomes a harder matter to deal with.