Hacker News new | ask | show | jobs
by deathanatos 139 days ago
… seems like we the HN community should find a new site to mirror with.
2 comments

There isn't one. As far as I know, no one really knows for sure how they bypass all these paywalls. (Most credible theory I heard: They actually just pay for the subscriptions.)
Many sites including Bloomberg have evolved such that even archive.today don’t have the full text of any articles. They’re doing no giveaways whatsoever.
Ghostarchive does a decent job for the same sites in my experience: https://ghostarchive.org/
Update: hmm seems like they're involved in this whole thing too somehow, how strange:

https://news.ycombinator.com/item?id=46629646

That comment is weirdly confusing/confused. But if you try archiving any site on ghostarchive, or clicking on any existing ghostarchive links, it just says "site is down for maintenance".

For now I've given up on using any archiving sites until we can find a safe and reliable alternative.

Most paywalls just allow search engines to read their content just fine. Because they do want discoverability, they want their cake and eat it.

There's a few publications that don't even do that though and archive.is is very good at bypassing them so I do imagine they use logins for those, but for the masses of sites it's not currently necessary.

You can't impersonate Google. Sites check the source IP and they don't overlap with Google Cloud.
Google isn't the only search engine in the world of course. It probably is pretty much the only one that matters in America but the world is not just America either.
It's the only one websites don't block. That's one reason it's so hard to make another search engine.
You can for sites that can't afford the cost of keeping up-to-date with the Google IP list without which they can lose timely indexing. That is many.
What do you mean by “afford the cost”? The list is free of charge (https://support.google.com/a/answer/10026322?hl=en-GB) and maintenance can be fully automated.
I mean cost of server setup and execution.
Then why hasn't anyone built a client-side browser addon that impersonates a suitable search engine?
They have. It's called bypass-paywalls-clean . It works pretty ok.

It just keeps getting banned from the addon catalogs because of complaints from media. The Firefox one was taken down by a french newspaper. So you have to sideload it, which is hard to do on Android.

Edit: it looks like even the github was taken down now: https://github.com/iamadamdev/bypass-paywalls-firefox

But yes it exists. And it works for most sites. It's just hard to get it now.

It's on gitflic.ru now.
Hmm yeah but their adversaries did achieve their goal by pushing it away from the mainstream sites. Now we're into this situation of "how much do I trust this vague Russian site with my browsing activity".

At least the addon declares the sites it's for and ignores the rest but still I'm a lot less comfortable with it. It's more something I'd install in a container now, limiting its usefulness :(

In practice I just use archive.today now.

What's your problem with that theory?
Has people's ability to read messages and formulate sensible replies been going down of late? I see this kind of meaningless replies more and more often these days.
Yes, there's a global intelligence crisis, due to tiktok instagram et al
Meaningless? Its a clear question.
You're accusing him of having a problem with it, which his comment does not imply.
None
I think there are multiple hurdles that make a new competitor very unlikely.

The first one is money. You need lots of it to run such an operation (servers, IPs, paying to bypass all these paywalls, etc.).

The second one is the legality, as no one wants to be hunted by the FBI, especially not for running a website that is also losing money.