[losteric]

> Plain headless Chromium is easy to detect by websites with anti-bot measures. Plain headless Chromium avoided getting blocked by websites only 2% of the time, according to our stealth benchmark.

> Our browsers avoid blocks 81% of the time on our stealth benchmark, and 84.8% on Halluminate BrowserBench, the highest of any provider.

Seems very unethical, no? Who uses service providers like this? The whole point of anti-bot measures is to get rid of bots - you are not wanted there.

These kinds of services inevitably make the web more human-hostile and expensive. Websites will continue pushing back on automated usage, meaning more hurdles to access content.

No doubt part of why we see this push for verified ID on the web - not just age gating and "protect the children", but also protect sites from bots, and protect ad revenue (not a statement of support; just seems like an obvious higher order effect)

[baby_souffle]

> Who uses service providers like this?

I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. I always set up price monitoring for any big ticket item I'm considering like appliances so I can see how their pricing changes over time. I also use scrapers for websites that don't have an API. I like having all of my purchase history indexed in a database where I can do analysis.

> These kinds of services inevitably make the web more human-hostile and expensive.

I would rather not have to spend more time circumventing stupid bot detection things. I would be more than happy to pay for access to some of this data that I cannot access any other way.. but sure, let's keep burning resources on a cat and mouse game that scrapers will always be able to win.

[pwdisswordfishs]

> I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS.

Have you considered offering, as penitence, a public feed to share the information that this process produces?

[saltmate]

Did you ask them for an RSS feed? Lots of people are pretty reasonable for such requests if you write a nice email.

[baby_souffle]

> Did you ask them for an RSS feed? Lots of people are pretty reasonable for such requests if you write a nice email.

Yep!

They're busy people or just don't feel the need to do anything beyond hit the "publish" button on their CMS and call it good and that's fine / why I have a robot to make an RSS for me :).

[saltmate]

Too bad, lots of CMSs have that built-in, probably a checkbox away.

[arianvanp]

The litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ out of the box or not

They do not.

[baby_souffle]

I don't know who 'they' is here but that's not the point? I would bet that a decent chunk of scraping happens because there is no API (or other machine-focused interface, like RSS).

"pay to crawl" sounds like the absolutely laziest possible way that a particular site could bolt on an API.

[mannanj]

has anyone been using this for success? wondering what kinds of pays they are getting. or are the crawlers just avoiding those sites.

[inigyou]

"we don't negotiate with terrorists"

[devmor]

This attitude, and by proxy this business are the epitome of selfish entitlement.

You state that you believe you deserve access to others’ resources, at their cost, despite their clear attempts to stop you from using them, simply because you want it.

[fc417fc802]

I'd counter that your attitude is a techno-authoritarian one. Why should anyone have any say over how I access and use a publicly available resource? At least so long as my actions don't directly cause technical problems for the service operator.

[saltmate]

> At least so long as my actions don't directly cause technical problems for the service operator.

That's the point of the criticism. The praise of their anti-anti-bot features reads like it is commonly used to cause technical problems to the service providers, be it intended or accepted for the cause.

[pocksuppet]

Anti-bot features are definitely used to cause technical problems to service providers you don't like.

[devmor]

> At least so long as my actions don't directly cause technical problems for the service operator.

But they do.

The reasoning you’re describing is not altruistic. It’s the same reasoning used by every AI scraper.

It’s the very reason I am paying a couple hundred dollars out of my own pocket every month to keep the websites of hundreds of small businesses and hobbyists online while I try to help them move to bigger cloud hosts, when I used to turn a small profit from it.

[fc417fc802]

> The reasoning you’re describing is not altruistic. It’s the same reasoning used by every AI scraper.

I think that's bad faith on your part. Clearly AI scrapers are aware of what they are doing and simply don't care. The entire purpose of my including the bit you quoted there was to explicitly exclude that sort of behavior.

[inigyou]

Maybe if you weren't using expensive anti-bot solutions people wouldn't use expensive bots.

[devmor]

That’s a great theory, unfortunately it’s defeated by the fact that I didn’t need to use anti-bot solutions until I was charged for 38,000x my normal ingress traffic in a single month by bot traffic.

[hirsin]

Look, it wasn't _my_ request that made the server fall over, it must have been one of the other several thousand thoughtless scrapers running on the website that caused it to die.

[fc417fc802]

If you're claiming that the operators of high volume AI scrapers that wantonly disregard rate limits and all common sense are unethical then I'm right there with you. But that's not at all what was described upthread nor is it the only way in which bots get used by any stretch of the imagination.

As far as anti-bot countermeasures go I quite like proof of work solutions since those disproportionately impact high volume scrapers without noticeably impeding a small hobby project.

Unfortunately the operators of many major websites appear to want something akin to DRM with the excuse of bots used merely as window dressing.

[ssl-3]

There was a time when a person could walk through a few department stores every week (or even every day) just to take note of some prices along the way, and ultimately tabulate them to try to identify and snatch up the best deal once it happens.

And if everyone did this, it'd be a real problem. The stores would be clogged up by geeks writing notes in little books with Parker Jotters and just basically wasting space and taking up air conditioning while they sleuth out the best way to put the screws to the company for a few measly dollars.

That'd be awful.

But not many people ever did that in stores, and not many individual people are doing that today with the web. It's really not a problem.

(And if a website in 2026 can't stand the burn of several thousand personal scrapers that are operated by people who actually want to buy stuff from it, then maybe that system simply sucks and needs to be rethought.)

[baby_souffle]

> There was a time when a person could walk through a few department stores every week (or even every day) just to take note of some prices along the way, and ultimately tabulate them to try to identify and snatch up the best deal once it happens.

This is how it started! I noticed certain things during my weekly shop that I did a double-take on and thought "wasn't that $cheaper last week!?". Took me ~ 45 min to figure out that the retailer actually has a really nice graphQL endpoint that powers the "view your previous receipts" function on their website. Of course they don't document this / make it available for 3rd parties... so scrape it is!

I wrote a bot to dump every receipt into a sqlite DB and I fire it up ~ weekly to pull down receipts that it doesn't have locally.

Turns out, not _everything_ has gotten more expensive @ my local grocery store over the past few years... just most things have :/.

> But not many people ever did that in stores,

There's a cottage-industry of firms out there that get gig-workers to pop in to $randomStore and take a picture of $randomItem on shelf w/ the price tag in the photo. The firms sell this info to stores that want to know how a competitor might be doing pricing / placing certain items on the more valuable shelf spots.

> and not many individual people are doing that today with the web. It's really not a problem.

That's my point! I scrape a few hundred pages per day across _many_ domains. My bots respect 429s and they have some other backoff/random-jitter strategies baked in to _not_ be the reason anti-scrape proliferates.

[irjustin]

Personally I consider it fair game in "price wars".

Dynamic pricing designed to extract every penny out. Then why shouldn't I be allowed to monitor your pricing changes?

[kjkjadksj]

You can already access these resources. What does it matter if you do the clicking or you have headless chrome do the clicking while you make a cup of coffee?

[baby_souffle]

This is my entire point!

And it’s why scrapers will always win; absolutely worst case, I get a screenshot of the content and have to process it further.

[Banditoz]

Dude, it's a web request. It's not that deep.

[devmor]

Dude, as someone who runs web servers, my pockets are not that deep.

I’m struggling to keep the websites of hundreds of hobbyists and small businesses alive right now because of people like this.

[inigyou]

How many requests per second are you getting?

[giancarlostoro]

I mean, this is how Google was built.

[cryptonym]

Not a fair statement. Google wasn't built on bypassing bot protections.

Google is providing a service to the websites they crawl.

They try to not crawl when we don't want them (robots.txt, clear user-agent, no-index no-follow...).

[giancarlostoro]

> Google is providing a service to the websites they crawl.

Yeah they're building an LLM and making it pointless to visit the websites.

[cryptonym]

Let's agree on "Google was providing a service". Current and future state can be questionable.

Btw you can still block it.

[mikeocool]

Whether or not scrapping publically available websites is unethical is probably up for debate. In some cases at least, courts have found it to be legal, even when the site is throwing up technical barriers or issues cease and desists.

What is likely unethical is the fact that they offer residential proxies. The residential providers of those proxies are frequently not aware they’ve been opted in to provide such a service.

[eab-]

> courts have found it to be legal

≠ ethical

[OhSoHumble]

I don't know if this falls under good ethical behavior but the one time I needed to mass scrape sites was to build an affordable housing directory for the Bay Area. Whether or not a unit of housing was under rent control was only available through private entities or apartment hosting sites and it was scattered and hard to find.

In my opinion, a directory of subsidized housing should have been provided by local governments and not through a plethora of real estate websites.

[inigyou]

Some are aware. I know some providers that pay you to run a proxy, but the pay isn't very much.

[embedding-shape]

> Seems very unethical, no? Who uses service providers like this? The whole point of anti-bot measures is to get rid of bots - you are not wanted there.

Unethical just because it does something someone else doesn't want? I guess it depends on why and what the intention is. I don't have time to sit 24/7 in front of a computer to get a ticket to some events, does that mean it's unethical for me to use my own bot so I can purchase a ticket to bands I'm a fan of? Probably not. But if I did so for scalping purposes? Then yeah, I'd agree it's unethical.

The whole point of anti-anti-bot measures is to be able to do things even if others don't think that thing should be automated, so from the hacker news audience, I think quite a lot of us have at one point or another engaged in stuff like that. Doing so merely for profits of course stinks, but for you to be able to have a fighting chance against scalpers? Probably OK.

[mystifyingpoi]

> even if others don't think that thing should be automated

It's an interesting thought that can be further explored. Could anything that's considered "unwanted" by a third party considered unethical, if I do it anyway?

If the hotel self-service restaurant has a sign "don't take the food out" and I take 1 apple in my pocket for a snack, is it unethical? Or maybe the sign is just for people that would otherwise take $100 of watermelons out of the cantina daily and try to resell it on the beach.

[Nathanba]

We ("society") do things to people that they don't want all the time, namely punishing them via the legal system. Then there are things that other people are doing that are immoral and should be punished for even though they are not illegal. And the whole class of inactions where we don't e.g buy something because it's overpriced and that's certainly something the seller doesn't like.

[joatmon-snoo]

An example I ran into recently: I wanted to scrape pricing data for used cars, to better inform a friend's decision about what to purchase.

I know there's a relationship between mileage and depreciation, but wanted to have a better sense of what that relationship is to know whether a given car was over or underpriced.

Similarly, if I was pulling that data to build a service of my own to offer to users... is that unethical?

[sroussey]

All of these questions are easily answered by the question: can I run the bot on the same PC I use regularly? If so, then do it there. If not, then don’t do it at all.

[pocksuppet]

This is often really good for your bot, because anti-bot providers are loathe to block what, as far as they know, is a residential CGNAT address. Sometimes you get more success scraping from home with Firefox or Chrome, than with an army of proxy networks.

[fc417fc802]

Why should technical capability to evade countermeasures dictate whether or not something is ethical? My view is that scraping remains ethical as long as your actions aren't causing technical problems for the operator. If anything, a retailer attempting to hide pricing data is what's unethical in my view.

[adolph]

> scrape pricing data for used cars

Time was you could get lovely json feeds from every site by iterating the inspector curl statement. Now-a-days you can't even use Selenium without Cloudflare getting grouchy. Last fall had to make my spreadsheet like a cave-person control c, control v. It wouldn't be so bad if the dealer aggregators' coverage was xor, but you have to dedupe listings. Then there is the whole online salespeople who don't show up at the dealership.

[inigyou]

There's a JavaScript property called navigator.webdriver that returns true if selenium is in use. Obviously, every antibot system checks it. Obviously, you can patch it to always say false.

[fc417fc802]

> But if I did so for scalping purposes? Then yeah, I'd agree it's unethical.

Is scalping actually unethical though? Sure it's unpopular, but I'd argue that's just your average person not properly grasping supply and demand and thinking through the consequences. If you want to sell something below market then you should raffle it off and take extensive measures to prevent transfer of ownership. The current practice is trying to pretend it's an open market while fixing the price, then getting angry when the obvious consequences materialize. Scalpers are merely the agents that correct a market inefficiency introduced by a dysfunctional status quo.

[saltmate]

Noone thinks concert ticket sales are an open market tho. They have one seller that sets the price, no competition between different sellers.

It only becomes a "market" after scalpers buy all contingent to resell.

[darkwater]

Unbelievable how somebody could defend scalping. There is no ethical or moral value in that practice outside of "I can earn money with that".

[fc417fc802]

I'm not defending the overall arrangement, simply pointing out that the blame is misplaced. If you want to sell below market value in a capitalist system then you must take appropriate measures to prevent a market from forming. More or less by design if one can form then it will.

Similarly I do not attempt to blame the rank and file employees of the ad tech industry for the actions of their employers, nor of the defense industry for the actions of the government. As an individual living under such a system either you move to make money when the opportunity arises or someone else will instead and you will lose out.

[darkwater]

> More or less by design if one can form then it will.

By this logic we should accept that the stronger beats and kills the weaker because it's literally natural.

[inigyou]

This is called "victim blaming". You are saying the blame for a problem shouldn't be on those who directly caused the problem, but on those who failed to prevent them from causing the problem.

You're right but in a different way. Scalpers aren't independent, they work for the artists to maximise artist revenue while absorbing the PR hit themselves.

[inigyou]

Scalping is ethical if and only if you are marketpilled. If you think anything outside the market matters, such as maximizing human enjoyment of the concert, then it's unethical.

[turtlebits]

Its unethical because you're intentionally bypassing restrictions. Just because others do it doesn't mean its okay.

If you saw a sign in a store that said "1 per person" or "for registered guests only", would you ignore it?

[windexh8er]

Look at what Google's doing right now with Chrome. On June 30 Chrome will remove the last flag that let uBlock keep working, and there's no workaround. Google says it's about security and performance, but is it? $239 billion in ad revenue last year seems to be the motivational factor. The "restriction" is a rule written by the company that profits when you can't block its ads, dressed up as protecting you. But... CISA recommends ad blockers as a defense against malware spread through ad networks.

The rules aren't always right and sometimes have unintended consequences. I think a bigger issue than Browser Use is all of the copyrighted material in every LLM. Given that precedent has been set with zero legal consequences, I'm not sure there's much of a leg for you to stand on here.

[embedding-shape]

> Its unethical because you're intentionally bypassing restrictions

I'd still consider why the restriction is there and why I'm thinking of breaking it, before deciding if it's unethical or not.

It depends, basically. Generally I follow the rules and restrictions, but maybe see them more as guidelines or suggestions.

[kube-system]

There are many ethical reasons to bypass restrictions. Colloquially, we just call them exceptions.

There are many valid ethical exceptions for evading anti-bot detections. For example: you are a white hat actor scraping a black hat site. There are hundreds of other plausible examples.

[inigyou]

If the sign says 1 per person, the reason it's unethical to take more than 1 isn't because you're disobeying a sign - it's because someone else might not be able to get one, and the sign is indicating to you this is likely to be a problem. If the store is about to throw out all the unsold ones in 5 minutes, then ignoring the sign is completely ethical.

[orf]

Was Rosa Parks unethical for sitting down on a bus?

The point is that the context matters: both the users context and the context of the restriction. It’s not as clear cut as “ignoring restrictions = bad”.

The restriction itself can be unethical, in the same way that bypassing a restriction can be unethical.

[Natfan]

we need a new version of Godwin's Law after this comment.

orf's law: > As an online discussion grows longer, the probability of a comparison with Rosa Parks approaches one."

[orf]

> As a discussion regarding if it’s ethical to ignore restrictions progresses, the probability of someone bringing up a famous case where someone ignored unethical restrictions approaches one

Seems reasonable to me. Substitute Rosa parks with another example of unethical restrictions if you wish - there are many.

[pocksuppet]

Do you think it is a problem that someone said it's always unethical to violate a restrictions, and someone else brought up Rosa Parks?

I propose a new law myself: as an online discussion gets longer, the probability of someone trying to defeat an argument by stating that it mentioned Rosa Parks or Hitler without engaging with the substance of that argument approaches one.

[BoorishBears]

Woah now, I'm for headless browsers but let's not start comparing any of this to Rosa Parks lol.

The reality is a lot of interesting, trivially harmful to non harmful things are illegal and we still do them anyways.

[jamiequint]

You're confusing law with ethics, they are not the same.

[choilive]

We use headless browser providers because the companies we interact with don't and won't create a proper API for us to use. Lots of legacy web apps/portals. Saves thousands of man hours.

[skybrian]

What do you think of Anubis and Cloudflare? If they block your bot, is that unethical?

Seems like doing business with other people should normally be based on mutual consent, not whatever you can get away with technically.

[inigyou]

Mutual consent to what? I didn't agree for rich people to own all the housing around here, is it mutual consent if I agree to pay one to get some, or is it extortion?

[skybrian]

Yes, when you buy a house it’s usually because the buyer and seller agreed to it. It seems better than, say, a foreclosure or an eviction.

[wnevets]

> Who uses service providers like this?

People who don't want their headless browser to get blocked?

[nateb2022]

> Seems very unethical, no? Who uses service providers like this? The whole point of anti-bot measures is to get rid of bots - you are not wanted there.

I'm familiar with companies automating access to software only accessible via the web with poor/no API support. This is software they pay (usually a lot of money) for, and usually has built in captchas to guard logins. They aren't a large enough customer to ask the removal of these captchas or whitelabelled (just one out of many SaaS tenants), so they simply work around that restriction.

[mystifyingpoi]

> Seems very unethical, no?

I don't think one can judge it ethically without considering the context. Are we talking about mass automated scraping? Or are we talking about me trying to get a good deal by scraping local used car dealership listing once per day for my personal need (just so I don't have to do it manually)?

One of these is strictly more ethical, but both will be blocked by Cloudflare for example. I'd happily use such service in my personal case.

[neilv]

I didn't use this company, but there are some legitimate purposes for scraping.

For example, at a startup a few years ago, one of the many technological things we needed to do was to monitor marketplaces for suspected counterfeit and contract-violating gray market goods for ~100 brands. And we couldn't just ask for data feeds, because, well, the marketplaces make money off of all those sales. And the off-the-shelf third-party data solutions were useless crap quality, worse than your average vibe-coding. So I made a bespoke crawler that gently and accurately tracked the data we needed, including global geofencing. So gently, I never got a whiff of disapproval or countermeasures (like throttling, 403, nor data poisoning). We were putting insignificant load on the marketplaces, for the purpose of helping to make the market better for both consumers and legitimate businesses. It was like a single "secret shopper" unobtrusively walking around some parts of a store. (And I also made an iOS app that did something different for actual secret shoppers in physical stores, for legitimate supply chain traceability for customers' brands.) Personally, I love the marketplaces, and hate the counterfeits, and this was my version of PG's advice that startups should be a little bit naughty.

Two of the problems with the current AI scrapers, which are destroying servers, and inviting backlash:

1. The gold rush situation brings out many of the crappiest people in the world. And also many who aren't crappy might behave in a crappy manner. (The latter, maybe because they're just emulating what they see, or extrapolating from the ethical temperature of prior industry norms, like surveillance capitalism in everything.)

2. Many of these scrapers are shockingly bad at what they do, and grossly inefficient. Almost like they're just pounding the same unchanging resources to DoS the servers for competitors. Or to drive sites to a protection racket company that's set up so they can also monitor cleartext. Or (Occam's Razor) just plain bad at what they do, and the people who pay for the salaries and computer resources either don't know or don't care.

[dagi3d]

Obviously don't know what percentage represents "legit" use cases vs other more morally questionable, but in our case we have a cms where content team can include external links and we need to verify periodically whether those links work or not, which is not as easy as making get requests with a client.

[brikym]

There are a lot of legit uses for keeping tabs on information. Price comparison websites for example allow the public to be better informed and fight hostile pricing strategies that are more common now corporate consolidation is at all time highs. Oligopolies don't want their prices scraped so they put up anti-bot measures. When the price history is layed out in a plain chart it becomes clear how efficient the economy is at segmenting markets and emptying wallets.

[MayCXC]

I built a similar system for an identity protection service that automated removing PII from directory websites like whitepages. Which was less ethical, stealth browser automation or monetized privacy invasion?

[sillysaurusx]

(I haven't tried this out yet.) My use case would be to take a snapshot of each HN story. This is surprisingly hard, because most websites prevent bots from doing that.

For example, Claude has a lot of trouble reading HN's front page. HN itself is fine, but the moment you ask it to pick out an article, it often chokes. The website has put up a verification captcha, or it's a paywall, etc. Paywalls can be bypassed by reading HN comments and looking for archive links. But those archives often block bots too, so you're back to square one.

Whether it's unethical is an interesting question. I believe I should have the right to do what I want with internet content, as long as I'm not abusive. Merely having a bot isn't abusive. It would be one thing if the bot is hammering a server or vacuuming up training data, but having a bot at all is presently very hard.

This service caught my attention because it could potentially solve the problem I'm running into. Simply taking snapshots of articles that hit HN shouldn't be so hard, but it is. HN sends millions of views to websites; one bot taking a snapshot isn't going to make a difference. I don't think it counts as "unethical" just because we're going against the website owner's wishes. When you post content to the internet, you sign up to share that content with everyone, other than what's denied by robots.txt. If it's not blacklisted by robots.txt, it should be possible for well-behaved bots to access.

I don't expect very many people here to care about the poor bot creators. Most of the bot creators are malicious anyway. But I personally lament the loss of being able to write a program that can process information from the browser in arbitrary ways. You should be able to, yet we're buying into the notion that it's okay for website owners to say "this content is only accessible by approved bots like Google, and everyone else can sod off."

HN proves it doesn't need to be like that. It gets dozens of millions of page views a day, a lot of which is bot traffic. HN only uses captchas for creating accounts or logging in. You're free to scrape any content as long as you respect the crawl delay of 30 seconds specified in robots.txt, and don't try to visit links that perform actions a human would take (like adding things to favorites or voting). That's how the internet should work: just deliver content.

[dist-epoch]

> one bot taking a snapshot isn't going to make a difference

until half of HN users start asking their agent to do the same, to summarize the top HN articles every day

[Dylan16807]

I don't think half is a realistic number, and any realistic numbers are not going to change the server load much. The real cost of that scenario is all the AI compute.

[Natfan]

no, to dang the real cost would be bandwidth

[inigyou]

Did that happen?

[inigyou]

It's not unethical to do something someone else doesn't want you to do. Discord doesn't want me to use it without buying Nitro, but I do. My bank doesn't want me to get a cheaper loan from a different bank. YouTube creators don't want me to use SponsorBlock.

[Banou]

Like if cloudflare and other protection services don't make the web more human-hostile blocking un-approved browsers and tools like cURL. Captchas and others already get solved by AIs nowadays, they're just friction done to collect labeling data for free from users...

[pocksuppet]

Which won't work, obviously, because as a bot operator, I'd just have my users provide their own IDs, or run another website to harvest IDs.

[ge96]

I briefly tried to do his job where it was scraping steam for CS GO skins (think a knife skin for $2,000.00) and yeah trying to find proxy poviders/get around the ip limit... tough one but market for it people paying for the tool (not mine).

[figmert]

Antibot measure also block real users at the slightest change they don't like. Anti-fingerprinting measure? You're a bot. Adblockers? You're a bot.

[stogot]

I wish simpler bots existed for consumers. I want to know when someone replies to me, when a price drops, when airlines open new seat reservations, when a new seat opens for a college class, when a concert is coming to my area for a musician I listen to, when my local grocer has new stock, when a new Hyatt offer is available in a city I want to visit, etc. doesn’t mean I’m abusive. I can have it check once a day. In almost all those cases, I want to spend money with the business but I don’t want to manually check

[__alexs]

There's no ethical consumption of... ad supported content.

[hollerith]

The people who've been in charge of the web (i.e., mostly the browser makers, but also the owners of the most popular sites) have made decisions that are IMHO severely anti-user. Although these anti-user design decisions have been accumulating for 30 years, users have had no alternative because all the content was on the web with way to get it other than to visit web sites with a web browser.

Now that there is an alternative (namely AI) people (including me) are flocking to the alternative. You want frame this as unethical bots versus ethically-acceptable human site visitors, but the main motivation for the use of scraping bots these days is to provide services (i.e, AI-based question answering) that users (like me) consider far superior to going directly to web sites for information because visiting web sites with a web browser is a frustrating tedious experience.

[phantomathkg]

Does it means the Wayback Machine is also unethical to you as well?

To me archiving the internet is way more ethical than putting bulk of the content behind paywall.

[cryptonym]

Coming with proper user-agent sounds ethical. https://archive.org/details/archive.org_bot

Author/publisher are owning their content. Expecting work of others to always be free doesn't sound really ethical.

[cute_boi]

Exactly these crappy companies like browser use is causing more captcha etc.. All these scraper companies should've been regulated heavily. They use residential proxy creating incentive for hacking IOT devices etc..

[inigyou]

The captchas are caused by captcha companies, nobody else

[ranger_danger]

Web archival/preservation services/projects that need to get past captchas and other bot checks are a prime target for a service like this... but I think their main customers are people just mass scraping parts of the internet for less altruistic reasons.

[zuzululu]

Once again I'd like to remind that violating Terms of Service isn't the same as violating some moral ethics. They are literally just expectations with no enforceable or legal boundaries.

For example I could write in my Terms of Service that you do not view more than one page on my website and expect you to send me a written permission to read the rest. I don't expect anybody to follow and I sure don't think less of those that do.

The push for verified IDs is not related to this, its more of a politically motivated attempt at selling fear to justify more surveillance.