Hacker News new | ask | show | jobs
by wolvoleo 139 days ago
Most paywalls just allow search engines to read their content just fine. Because they do want discoverability, they want their cake and eat it.

There's a few publications that don't even do that though and archive.is is very good at bypassing them so I do imagine they use logins for those, but for the masses of sites it's not currently necessary.
2 comments
direwolf20 139 days ago
You can't impersonate Google. Sites check the source IP and they don't overlap with Google Cloud.
link
wolvoleo 139 days ago
Google isn't the only search engine in the world of course. It probably is pretty much the only one that matters in America but the world is not just America either.
link
direwolf20 138 days ago
It's the only one websites don't block. That's one reason it's so hard to make another search engine.
link
chrisjj 139 days ago
You can for sites that can't afford the cost of keeping up-to-date with the Google IP list without which they can lose timely indexing. That is many.
link
otterley 139 days ago
What do you mean by “afford the cost”? The list is free of charge (https://support.google.com/a/answer/10026322?hl=en-GB) and maintenance can be fully automated.
link
chrisjj 139 days ago
I mean cost of server setup and execution.
link
otterley 139 days ago
The server that is providing the content exists already. That's a sunk cost.
link
chrisjj 139 days ago
"setup and execution".
link
mr_mitm 139 days ago
Then why hasn't anyone built a client-side browser addon that impersonates a suitable search engine?
link
wolvoleo 139 days ago
They have. It's called bypass-paywalls-clean . It works pretty ok.

It just keeps getting banned from the addon catalogs because of complaints from media. The Firefox one was taken down by a french newspaper. So you have to sideload it, which is hard to do on Android.

Edit: it looks like even the github was taken down now: https://github.com/iamadamdev/bypass-paywalls-firefox

But yes it exists. And it works for most sites. It's just hard to get it now.

link
eipi10_hn 139 days ago
It's on gitflic.ru now.
link
wolvoleo 138 days ago
Hmm yeah but their adversaries did achieve their goal by pushing it away from the mainstream sites. Now we're into this situation of "how much do I trust this vague Russian site with my browsing activity".

At least the addon declares the sites it's for and ignores the rest but still I'm a lot less comfortable with it. It's more something I'd install in a container now, limiting its usefulness :(

In practice I just use archive.today now.

link
eipi10_hn 138 days ago
Yeah it's unavoidable. Bypassing paywalls is not a good idea for tools that depend on browsers' stores distribution.
link