Hacker News new | ask | show | jobs
by benjiro29 8 days ago
As somebody who did plenty of scraping for his own little projects, lets just say that reddit their security concerns are just PR for "we do not want to keep supporting old.reddit".

While yes, you can not simply scrap new reddit as easily as a pure html scraper is cheaper to run. And while the new reddit its js "slow down" scraping as it need to run over a headless browser. Do you slow scraping down a lot?

No. Because you can simply spin up more instances and route over more proxies. Given the fact that you can scrap millions of pages per day, on a single cheap 1 CPU node...

The limiting factor for scraping a website is not html vs js / client vs headless browser. Its avoiding detection by running from different ips, faking browser information and masking your traffic to not give off the smell of automatization (and avoiding tls fingerprinting).

I wrote this type of stuff before we had AI to help write it, in a few days time. Now with a LLM at your fingertips in a few hours, you will have a complete working version with cloaking build in.

And the larger size difference does not matter, you can block irrelevant files via faked cache responds, to give the server the impression that your a repeat client, and no need to flood your browser instance with images, fonts etc. Some think that this is a defense, aka throttling you with increased bandwidth usage lol

So in my personal opinion, its not about dealing with scrapers, but trying to push people to new reddit, so they can finally phase out old reddit.

PS: So much fun reading new reddit on your phone browser, to have it pop-up a "use our app" banner, that prevents you from interacting with the website anymore. And even blocks firefox its scroll ability (and thus accessing to the browser its nav bar). Tip for people: delete the sites cookies and cache regularly, to prevent this usage tracking, so the banner newer shows.

4 comments

New reddit is more annoying (to me as human, but also to scrapers) in two very important ways:

- old reddit shows more or less the raw comment tree, sorted, like HN. New reddit does aggressive folding. You get a couple comments for each branch and have to click "show more" to see more

- new reddit forces you to log in to view anything tagged NSFW, including entire subreddits. Old reddit does not enforce this, along with not enforcing some other restrictions reddit has introduced over time

Of course none of this stops a scraper. And there is still the API. But these things make it easier to detect scrapers because they have to make a lot more requests and have to handle user sessions that make them easier to track

As I understand it, anything publicly viewable is free to be scraped without issue.

If you scrape stuff behind a sign-up, you potentially open yourself up to being sued for breach of ToS.

Now I'm sure the Chinese labs don't give a rats arse about that threat, but I imagine it stops the likes of Google, OAI, Anthropic from scraping and instead forces them to purchase the data.

Not how it works ... Anything public viewable is still subject to a TOS. And the whole copyright or whatever still applies.

If your argument was valid, we can scrape news websites and show their content freely. No ... You get sued and lose the case.

See Google News that got their behinds in court and lost. Short summaries are allowed / transformative, but you can not just take content (even without a login). Not without getting into civil court if somebody wants to press the matter.

Now, if you scrap and never make that data public or transform it (LLMs). Then it becomes a harder matter to deal with.

I can't use new reddit - old reddit lets me disable infinite scroll

I think all major platforms should have an option to disable infinite scroll

Otherwise I look up and its 3 hours later

> the purpose of the system is what is does
> "we do not want to keep supporting old.reddit"

the second old.reddit goes is the second I stop using that site

it's already very obviously chock full of slop and AI bullshit, with some subs having long unavoidable lists of "your post was removed because you did not verify" trails that imply that automated posters found a thread and went hard

but add in privacy crushing bs -- no reason to be there anymore. Back to old school forums

I never used to be an old.reddit die-hard, but new got so bad it chased me there to where I now use a browser extension to redirect all reddit links to old. I was convinced that the website continued to get worse so they could steer you into the app which hoovers up all your personal information. But then the app is so godawful unusable that I think we're just looking at straight-up incompetence.

So yeah, I'm in the same boat: if they drop old, they drop me.

So, is it back to Digg, circle around to Slashdot, or is there a new kid on the block?
That solo dev who developed RES could have forked it and "stolen" the whole site if he wanted to. That would have been so epic.
If the “protocols not platforms” philosophy is the right track, usenet/nntp.
nntp was built on horribly naive assumptions that do not carry to the present day internet. nostr is a better option.
Digg tried to make a comeback, but was quickly smothered in the crib by AI bots.
I'm mostly just here now... More forums for other stuff would be nice though.
Lemmy is pretty good.
Fark hasn’t gone anywhere.
Old–school forums never left, still inhabited by the few boomers and genXers who could figure out the Internets back in the day and aren't dead yet.

The gamification of Reddit and the ephemeral nature of the main page along with subreddits with varying amounts of overlap contributed to more active interactions, though.

Some of the volume was lower quality but not all.

The boomer posts about his petunia breeding program on a BBS and he gets feedback half a year later after the other active user gathers the energy to travel to the computer room again.

On Reddit, maybe a zoomer would have asked if you can smoke petunias in Brazilian Portuguese, but that may still count as welcome human interaction to the old man.

None of that applies to new deaddit, though.

Forms are sadly dead. They cost a lot to operate and maintain security updates for so without a critical mass of users they are dead
Just like with VR, you can keep on claiming forums are "dead" all you like. Me and all my friends will keep on using them daily.

therearedozensofus.gif

"Good thing you're the majority of people, then." - samplatt [0]

[0]: https://news.ycombinator.com/item?id=49002685

I was a hypocrite even writing that comment; my latest phone ALSO has an FM radio. Guess that reddit-quippiness infected me too much that day, sorry.
They are dead the same way newsgroups are dead. Some local communities survive some might even thrive but in aggregate more are lost than created
How did vBulletin/phpBB came to dominate the forum space in the English internet? To me, that seem like the source of a lot of problems for the forum culture and centralization into Reddit.
I used to run a few forums. Did some work for SMF 1.x back in the day too.

The thing that got vBulletin to succeed over UBB was themes and customizations/hooks.

The thing that made phpBB popular was it didn’t require a license like vBulletin.

Honorable mention is SMF. It replaced YaBB SE, was also free, but had an amazing mod system that didn’t require any manual file editing. And [Unknown] and his team did a great job on the database keeping it way more performant than phpBB was in the free space.

vBulletin Did a better job with security. PHPbb kept getting trashed by spammers but they were the primary tools at the time for self hosting. The ease of customization and the availability of skins made it a big selling point
Some are alive, eg. LowEndTalk. They are not as general as Reddit.
They do not need to be.

They are one of those things that governments and big tech do not like. They are too easy for just anyone to operate, so I think we can see more legislation aimed at making running forums harder or more expensive, like the Online Safety Act in the UK.

As programmers we assume we have to preemptively comply with every possible rule, which is a heavy burden, but I think in every case that isn't completely egregious (like 4chan) the government asks you nicely first. When I look at people with successful projects they only seem to be concerned with following the most obvious laws and the ones they have been warned about following.

An exception is copyright law. Because of massive corporate pressure, you can go to jail quickly for sharing a copyrighted work. Fortunately there's still section 230 which immunizes intermediaries.

Cost a lot to operate? apt-get install forum-software. All security updates with apt-get updates. Use something like Debian, and you keep the same forum software for years with zero config changes, but with security updates.

I know endless local groups. Bike clubs. Hobby groups. They still all run on forums and mailing lists. No ads. No tracking. No doom-scrolling or this and that. Cost?

For many of them, a $5/month VM.

They are cheap to operate now. This is something governments and big tech will fix.

As I said in another comment they are no longer as cheap to operate in the UK because of regulatory requirements, and this sort of regulation tends to get copied and extended around the world. All for the sake of the children, of course.

How much will it cost with UGC requires age verification and round the clock moderation? When its a risk to do it without a company to shield you from liability?

Local bike clubs and hobby groups presumably know their members, so age verification and moderation aren't issues. Despite computers theoretically being able to connect to anyone in the world, you don't have to use them that way.
Someone is always hungry and willing to ignore international norms for a greenback. Vanuatu will sell citizenship to any hoodwinking criminal for a paltry sum. Comoros will sell internet "gaming licenses" to any money launderer with some change rolling around. Some other shithole island will happily not pass the law so that whatever warlord is in charge can buy enough coconuts for the plebs not to overthrow him, by selling pirate internet services. Then all the forums will be hosted there.
You have to moderate it
Seeing as people have been self-moderating some of these forums for more than 20 years, it's a solved, non-issue. Just because you're not familiar with this, doesn't mean it doesn't work. It's worked since the 70s with bulletin board systems.

Yes, those required moderation too.

Again, solved problem for community sites.

Reddit mods are volunteers as well.
Subaru forums are all somehow going strong
And yet here we all are.
the big click of reddit like yahoo forums and other earlier things is that each 'sub' reddit can have its own rules culture, etc. its like facebook groups in a way. Unfortunately with how bad reddit is, I see mostly my interest niche hobbies moving to facebook.
I used to use infinity for reddit, after the api saga, I was able to patch it with my own api key since personal ones were still free. I lost the key since, and had to re-install Infinity, so when I go to the reddit api page, turns out they don't give you keys any more, and instead you need to sign up to their developer platform, which I requires a lot of hoops to get a key for free, and from what I understand, that's a big if.

Anyway, needless to say I've stopped using it for the past 3 months or so. I still open old.reddit on my computer every now and then, but I use reddit so much less because of this.

They started revoking the old personal api keys anyway.
It's also like the only place left on the internet to reliably find real people reviewing things with real opinions.
With AI posts and karma farming -not to mention good old-fashioned astroturfing ...how reliable is that, really?
and yet, it remains, in many subreddits.
I've almost stopped using it because 3rd party apps finally died a couple months ago. Old.reddit isnt too ergonomic on mobile and new is just awful. Seems like a lot of that sentiment here, they might just kill a big part of their userbase before IPO!
>Back to old school forums

It's the year of Our Lord 2026. Good luck discovering forums that you don't already know about!

> So much fun reading new reddit on your phone browser, to have it pop-up a "use our app" banner, that prevents you from interacting with the website anymore

Hmmm, I remember this from the past, but one of my adblock rules probably catches this one, I haven't experienced it for years.

I always surf in anon windows that forget all cookies. Works since 2010 without any plugins. :)
Works a treat, I reckon -until the moment comes when you actually want to interact and/or post.