Show HN: A bookmarklet to remove clickgates on New York Times, Medium, etc.

[timar]

Recently i stumbled on too many clickgates on the Medium blog Towards Data science. Considering that most people publish to share knowledge on Medium and are driven into putting their content behind a paywall, without actually getting paid for it, including myself. I felt like Medium is running the academic publishing scheme. Get free content and get paid for it. So I decided to create a small script to bypass the paywall on Medium, it turns out it also works on other newssites. Heres the website: https://sugoidesune.github.io/readium/ For the curious I will explain the technical aspects in a comment.

[timar]

Here is a gist of the commented code. It works by fetching the HTML content of the website, anonymously with no cookies. Using the fetch API. In a second step the HTML is preprocessed, removing javascript, inserting elements like images that might be done through javascript etc. The third step is to rerplace the current windows HTML with the clean-preprocessed HTML with the article.

https://gist.github.com/sugoidesune/884bfdf8a975920e98e7307e...

[wiseleo]

Nice...

[hprotagonist]

https://medium.com/@tgaul/introducing-unobstruct-230e4e95cf5...

works nicely for iOS/mobilesafari.

[lewiscollard]

Good stuff timar - this definitely beats my trick of hitting the Esc key at just the right moment to stop JS from loading. I've gotten very good at that, but also life is very short and I shouldn't need to! So your bookmarklet will come to the rescue :)

Could this be integrated with Firefox's Reader mode somehow?

[timar]

Not very familiar with firefox or reader mode. But if it's something firefox does with the html of the website youre currently on. Than I see no reason why it couldn't work together. First getting the article with Readium, secondly starting Reader mode.

[barretts]

Do you think the 1,600 journalists at The New York Times, many of whom work in difficult environments (including the White House, lol), should work for free? Quite the Scrooge move.

[wolco]

Calling someone a scrooge or cheap for finding a hole in a technical limitation device that is sloppy in its implementation as it is click-bait in it's dark patterns of pretending the page is an open page/free page so it can compete in the search engines but in reality it presents a paid-gate that only appears after leading content is very unhackernews like.

NYT one of the largest most profitable companies pretends to show you an article baits you and demands money and you are a scrooge for not paying? In that context they are the rich entity and should give the content away (the way Scrooge should have). No where in the story does it mention Tiny Tim finding coal in Scroogle's garbage and someone stopping him and calling him cheap for not paying full price because Scrooge employes so many people.

[baddox]

> it is click-bait in it's dark patterns of pretending the page is an open page/free page so it can compete in the search engines but in reality it presents a paid-gate that only appears after leading content

This is the most important point here. If your content is not actually public, you really shouldn't get the benefits of search engine exposure, HN exposure, or even the distribution from sharing what appears to be a URL to a hypertext document.

[godshatter]

Whenever I see a New York Times link somewhere, I just move on now since I know I can't read it. I'm not going to pay a subscription for every web site I might want to read an article on. Their business model is not my concern; if they want my business they'll have to try something else.

[sjdegraeve]

For those who want a daily concise version of the New York Times, check out https://timesdigest.com/.

It's not free, but I've found it to be a great way to read the NYTimes.

Example of the 10-page daily PDF: https://timesdigest.com/samples/timesdigest

[cmdshiftf4]

>Quite the Scrooge move.

Seconded. There's something very tasteless about a bunch of very well paid tech employees who'll readily decry adtech, tracking and data brokerage (despite many of them working on it themselves), who'll proclaim "there has to be another way!", who'll also lambast modern journalism for "falling quality", circumventing content subscriptions and depriving publishers of revenue simply because of a built-up sense of entitlement to everything being free.

If the content is worth reading, i.e. it's from a publisher you believe to be of sufficient quality to try and find methods to work around their subscriptions, then it's worth paying for.

[antisthenes]

Reading many opinion pieces from NYT, I wouldn't cry if the vast majority of those journalists became unemployed. It's outrage journalism and thus not worthy of my time or attention.

NYT does do good journalism, but, even for a company with their credentials and resources, its share seems to be ever dwindling as time goes on.

[timar]

I feel like, if PewDiePie can make 1 Million a month in youtube ad-revenue and 8 million total. A Newssite like NYT ought to be able to support their staff, thanks to a global reach and 240Million views per month.

[sharkmerry]

8 million dollars over 1600 journalists is $5000 each. Pewdipie doesnt have the costs of running a news company and hes not in the news industry

[timar]

Yeah, thats 8 million a month. i feel like 5k a month is pretty good pay for a journalist.

[pvg]

NYT should adopt the business model of a youtube edgelord and then pay its journalists submarket wages does not sound like a very realistic approach to running a solvent news organization.

[tshaddox]

I don't think they should work for free. I'm not under the impression that they do work for free, or that blocking ads or paywalls will cause them to work for free.

[knolax]

They should pay me reparations for reading some of the nonsense they publish.

[barretts]

I'm sorry for your indentured servitude.

[dang]

Please don't feed egregious comments by replying to them. That's in the site guidelines: https://news.ycombinator.com/newsguidelines.html

[knolax]

Thank you.

[neeksHN]

Do you assume all HN users would reach the paywalls' read-limits through organic everyday use?

I felt forced to disable cookies and javascript on the following sites just after exclusively clicking on links submitted to HN: NYT, Medium, Washington Post, Bloomberg, MIT Technology Review and Harvard Business Review.

I'd prefer if submitters choose more user-respecting sources, but until then I refuse to feel guilty over stopping these intrusive sites and their dark patterns.

[larnmar]

No, I feel that they should all go and get other jobs.

[kerkeslager]

Do you think that if the New York Times sends you a copy of their article, you're obligated to pay for it, even if you didn't agree to pay for it?

I'm happy to pay for content. I donate $10/mo to both Mother Jones and ProPublica. That's more for each than the cost of a New York Times subscription. Neither of these has a paywall. I'd prefer that they didn't have popovers either, but that's about as good as I can expect in the modern anti-privacy, anti-attention culture of the internet. I just don't feel obligated to pay for content that is sent to me without me ever agreeing to pay for it.

I also think search engines shouldn't index paywalled content, and that newsfeeds like HN or Reddit shouldn't allow links to paywalled content, and I consistently downvote paywalled content.

EDIT: I'm also going to start requesting that people not link to paywalled content when I see it, I think.

[chiefalchemist]

Sidebar: working for and writing at tbe NYT - or any "news" publisher - doesn't make you a journalist. Journalist is a process and a result. We're too quick to use the word journalist. The low bar should not be so normalized.

[fidla]

Thanks! I've been looking for something like this for ages. I use Brave browser on my android and pc. This will be super handy!

[qnsi]

I though people are paid for posting on medium?

[timar]

They do, but you have to apply and be selected. Compared to all users, its a fraction. And everyone else gets pushed into the paywall, because medium basically says: "If you don't we will not promote it and your post will die in obscurity"

[sharkmerry]

so medium is running a business? Did you apply and get turned away?

[timar]

No, I don't aspire to be a writer for a living. If I publish something I do it to share knowledge or for personal branding.

[timar]

Ofc they're a business. But in youtube terms. As a creator you either put your video behind a paywall,and not necessarily get paid, or your video will never show up in any recommendations. So unless somebody searches for it or has the link. Nobody is going to see it. And every viewer on youtube gets 5 videos a month.

Has a bit of a bad taste for me.

[Eduard]

How to use it with mobile Chrome on Android?

[wiseleo]

Thank you. Both for the code and for giving me an inspiration. :)

[mchintu]

This is fantastic

[selfishgene]

This one is also quite helpful:

https://github.com/iamadamdev/bypass-paywalls-firefox

... for all of the folks out there who don't have a university picking up the tab for the articles that they want to read.

It's important to remember that paywalls disproportionately impact those who are not "working" in an academic setting.

[atombender]

For Chrome and Brave: https://github.com/iamadamdev/bypass-paywalls-chrome. Seems to work well.

[barretts]

And evading paywalls disproportionately impacts the journalists creating the stories you seem to value, though for some reason refuse to pay for.

[baddox]

Do you think that people reading newspapers for free at the library negatively impacts journalists?

[Red_Leaves_Flyy]

The average value I glean from an average article is less than if I had stared at a wall instead. I'll gladly pay a modest sum for a consistently high quality long form news service. However, everyone I've found is asking hundreds- thousands/yr which is neither modest nor affordable for most people, myself included.

NYT, wapo, BBC, fox, CNN, etc all push out tonnes of clickbait garbage every day. If that's not making them enough money then maybe they should get back to investigative journalism. I want long, painstakingly neutral, well researched, and cited articles on topics that concern me. E.g. healthcare, economics, politics. I find more value from the participants in discussion boards I frequent than just about every article I read. Sure, there's a few that provide more value than the participants like the work out of the ICIJ, but they don't hide their work behind paywalls, have ads, or run clickbait, and yet somehow they remain solvent... Hmm...

[LoSboccacc]

this it's literally facilitating copyright infringement

if you don't want to pay for them is fine, just don't read them

[baddox]

It's certainly not copyright infringement to download a hypertext document from a company's public web servers.

[LoSboccacc]

did I say it's copyright infringement?

no.

"facilitating" is not just there for shit and giggle.

https://www.law.cornell.edu/uscode/text/17/1201

[baddox]

What I meant is that the thing it's facilitating is not copyright infringement. It's not copyright infringement to download a document that was intentionally provided by a public server. The fact that they display paywalls to some visitors under some circumstances is irrelevant. Remember, you can also "circumvent" these paywalls by reading a newspaper at a library or office lobby.

[Red_Leaves_Flyy]

That's stretching the definition. By your argument any parsing of a payload containing copywritten work is copy right infringement.

[LoSboccacc]

no? section 3a circumvention definition is exceptionally (and intentionally) wide in the and section 3b does the same for what is considered a restrictive measure.

> any parsing of a payload

again, no? authorized parsing does not.

[Red_Leaves_Flyy]

As far as I'm aware most websites don't have (enforceable) eulas that state what parsers can and can't be used. If I want to read the binary stream from the wire that's my prerogative.

If you have a website that shows a blank page, to everyone that doesn't send a specific token to your webserver, and I discover that I can view said page by manipulating the dom with any old browsers console I am not violating your copyright. You willingly sent that information.

To bring this home, if I can read an article visually obscured by code that has otherwise been given to me in an unencrypted form I am not violating copyright. If a company has a problem with people seeing everything said company sends to them then they're woefully ignorant of how browsers work. The solution is to stop including the article. Rightfully, this kills seo rankings.