[diggan]

> it could present unacceptable risks for application developers or be used as a method for malicious attacks (e.g. credential stuffing or fake account creation).

The article seems to want to distinguish between "bad" and "good" bots, yet beyond the introduction, seems to treat them exactly the same.

Why are website authors so adamant I need to use whatever client they want to consume their content? If you put up a blog online, available publicly, do you really care if I read it in my terminal or via Firefox with uBlock? Or via an AI agent that fetches the article for me and tags it for me for further categorization?

It seems like suddenly half the internet forgot about the term "user-agent", which up until recently was almost always our browsers, but sometimes feed readers, which was acceptable it seems. But now we have a new user-agent available, "AI Agents", that somehow is unacceptable and should be blocked?

I'm not sure I agree with the premise that certain user-agents should be blocked, and I'll probably continue to let everyone chose their own user-agent when using my websites, it's literally one of the reasons I use the web and internet in the first place.

[mcstempel]

Hey there, I'm the author of the post. I'm actually pretty sympathetic to your viewpoint, and I wanted to clarify my stance.

I actually spent years working at a "good bot" company (Plaid), which focused on making users' financial data portable. The main reason Plaid existed was that banks made it hard for users to permission their data to other apps -- typically not solely out of security concerns, but to also actively limit competition. So, I know how the "bot detection" argument can be weaponized in unideal ways.

That said, I think it’s reasonable for app developers to decide how their services are consumed (there are real cost drivers many have to think about) -- which includes the ability to have monitoring & guardrails in place for riskier traffic. If an app couldn't detect good bots, that app also can't do things like 1) support necessary revocation mechanisms for end users if they want to clawback agent permissions or 2) require human-in-the-loop authorization for sensitive actions. Main thing I care about is that AI agent use remains safe and aligned with user intent. For your example of an anonymous read-only site (e.g. blog), I'm less worried about that than an AI agent with read-write access on behalf of a real human's account.

My idealistic long-term view though is that supporting AI agent use cases will eventually become table stakes. Users will gravitate toward services that let them automate tedious tasks and integrate AI assistants into their workflows. Companies that resist this trend may find themselves at a competitive disadvantage. Ultimately, this has started to happen with banking & OAuth, though pretty slowly.

[do_not_redeem]

It seems like cases (1) and (2) would both be better handled by letting the user give their user agent a separate security context if they choose, instead of trying to detect/guess what kind of browser made that http request. I'm thinking about things like oauth permissions, GitHub's sudo mode, etc. Otherwise your magic detection code will inevitably end up telling an ELinks user "sorry, you need to download chrome to view your payment info".

[mcstempel]

You read our mind! https://stytch.com/blog/the-age-of-agent-experience/

Very much agreed that's the long-term goal, but I think we'll live in a world where most apps don't support oauth for a while longer (though I'd love for all of them to -- we're actually announcing something next week that makes this easy for any app to do)

But we're also envisioning an interim period where users are delegating to unsanctioned external agents (e.g. OpenAI Operator, Anthropic Computer Use API, etc.) prior to apps catching up and offering proper oauth

[nonrandomstring]

What came up in this interview [0] was that

1) Because of "AI" we're moving more to API-like model in which the end user gets more say how they want to consume content.

2) That is in tension with (ahem) intention. We can't direct the user "experience" and have a "positive model" (not based on denylists). We can present data bit we can't enforce our intentions (informally defined ideas about how it may be used).

3) That means we must move to a behavioural security/access model in place of identity based ones (including categorical identity like ASN, user-agent, device type... )

[0] https://cybershow.uk/episodes.php?id=39

[rendaw]

Why can't users revoke permissions if the service can't detect good bots? Those seem wholly unrelated.

[baobun]

> That said, I think it’s reasonable for app developers to decide how their services are consumed

But it's a far step from that to (attempting to) control the user agent, or only allow blessed clients/devices.

Of course the site operator is concerned with limiting and preventing abuse by malicious users and agents, and an app developer should build for enabling that.

> Main thing I care about is that AI agent use remains safe and aligned with user intent

Nice and all. Keep a level perspective though: At scale, you can't keep control of your users not getting scammed/phished/hacked, or plain doing destructive uninformed actions on their own accord. Similar here: If you aim for 0, that will be to detriment to (at best, I believe) your growth.

I believe the kind of patterns you describe in the article are in fact anti-patterns. Look at the kind of web and internet they lead to. Look at what they do to individual agency in society. Across the board, abuse is increasing alongside negative side-effects from false positives of these kinds of counter-measures - which will invariably end up abused (by ignorance or intentionally) to exclude an increasing number of "undesireds". Systematic discrimimation is an apt term for the emergent consistent blocking of certain groups and individuals even if "it's just the stats playing out that way"?

Consider accessibility, and the diversity of humans. It is a folly to believe you can craft a singular user-experience that works satisfactory for everyone, or even catalogue and "officially support" what's in need by your entire target audience. By blocking access to screen readers and other accessibility agents you limit or prevent the use from those relying on these tools.

> My idealistic long-term view though is that supporting AI agent use cases will eventually become table stakes.

My optimistic long-term view is that accessing content on my own terms with an agent I compiled myself is still an option (without any need for dystopian centralized signing services a la apple/mozilla), and that companies are still legally allowed to offer that option.

[soulofmischief]

Plaid is not a "good bot" company. Despite posturing from leadership, it is fundamentally unethical to build a pervasive banking middle-man service which requires users to surrender their private account credentials in order to operate. What if every business operated this way? It's disgusting that companies like Plaid have considerably set back public discourse on acceptable privacy tradeoffs.

[randunel]

I'd assume they had to work with what was offered. As long as banks required usernames and passwords with no oauth possible, what's plaid to do? Their users wanted their service, but the banks used username password credentials.

In any case, "good bot" doesn't refer to best practices such as rejecting suppliers with antiquated auth and guiding users to others, it refers to not being intentionally malicious and acting as users' agents instead.

[clint]

You write as if someone held a gun to your head and force you to sign up for Plaid. Plaid doesn't require anyone to use it.

Your bank is the entity you're ultimately upset with, don't malign a company that generated a _very good solution_ to a _huge problem_ and THEN worked with their industry peers to cajole these huge banks to let you have access to your data how you want to use it. Before Yodlee and Plaid came around there was a snowballs chance in hell I could ever hope to get at my banking transactions in an API and now I can, and in many cases I never have to give supply my banking credentials to anyone but my bank.

[soulofmischief]

> You write as if someone held a gun to your head and force you to sign up for Plaid. Plaid doesn't require anyone to use it.

There is not a physical gun pointed at my head, but an increasing amount of digital online interactions are solely gated by Plaid. I've run into plenty cases where I simply had no choice, for example dealing with landlords.

And you already know how long it takes for financial systems to evolve once in place, as evidenced by your own frustration for them not embracing APIs and digital sovereignty. So once a solution like Plaid is in place, we're normalizing this kind of man-in-the-middle security nightmare for generations to come. Even if Plaid's founders did not have malicious intent, the company will eventually change hands to someone less ethical, and the door is open for other companies to seek the same kind of relationships with end users. If not malicious, Plaid is brazenly reckless and short-sighted.

And regardless... I as a consumer do not want to hand over my passwords to a man in the middle, I'm already angry enough at the security and password restrictions I encounter now with financial institutions. If I am in a position where I cannot rent a home or make an important purchase without interacting with a company like Plaid, where is my digital sovereignty?

[soco]

I think this anger with Plaid is unwarranted. Without them, or before them, you had zero API access because the banks (including yours) don't give a rat's ass on your fancy access needs. Now Plaid managed to gather together some kind of access. Are they to blame because they managed that? Do you still have any alternative with the bank? I think no, and no. You can get back to the "standard" situation of no API, no guns involved, or you can use them as middlemen. Or you can create your own middleman service if you like and everybody will appreciate your Plaid alternative (except Plaid, I suppose).

[BoorishBears]

I think it's warranted if you don't look closely, unwarranted if you look deeper, and once again warranted if you look even deeper...

Before Plaid there, the floor to require a bank account to do something in your SaaS was high to impossible.

Now the floor is low, and we got a bunch of applications that take advantage of that, so good right?

The problem is most of those applications are not in your best interests as a person.

-

It mostly just enabling a bunch of junk BNPL debt and modern day payday loan schemes.

It allows offerings that are too risky to be good ideas to patch things up by just peeking into an account and making sure they'll be able to take their $X before some other rent-seeker drains the account for the month.

It also normalizes so much more access and visibility than is actually needed, so even in cases where the risk was acceptable before, now why not just peek really quickly and improve your bottom line at the expense of having yet another service with access to your financial data.

Overall Plaid probably has not been a net positive for the average person. Other countries have open banking platforms but they're also must stronger on regulation and oversight than the US, so you don't seem it become quite as much of a negative.

[soulofmischief]

> Are they to blame because they managed that?

Do you understand that the ends don't always justify the means? Do you understand that not trading security and privacy for convenience means putting up with inconvenience? My complaints are warranted because yes, they are to blame, no, I do not want to be forced to use their service.

And when the company is eventually sold and financial transaction data harvested (whether against the wishes of the founders or not, loopholes exist), apologists will turn around and blame the new company instead of Plaid, who opened the door for them.

> Or you can create your own middleman service if you like and everybody will appreciate your Plaid alternative.

I think the financial tech market is rapidly evolving, and I'll just wait. If I need financial automation and a service like Stripe is not available, I can always use a cryptocurrency which respects my autonomy and privacy.

[immibis]

Well then banks should offer a proper API with tokens and permissions.

What's that? They don't? Guess I'll just have to give Plaid my password then. Stupid banks.

btw this is the exact same way Facebook got people to migrate off MySpace.

[cudgy]

So do you also expect the bank to back you up if you get hacked, given your exposure of the password to plaid or other services?

Not sure banks are the best example for this discussion, though, since banks have legitimate reasons to secure and promote security of their accounts that is beyond simple IT resource usage.

[soulofmischief]

I remember Facebook's shady user acquisition tactics, and I also do not use Facebook and similarly think their business model is morally bankrupt.

> Guess I'll just have to give Plaid my password then.

Learned helplessness, trading digital sovereignty for convenience. There is a larger war being fought here that is bigger than you or me. Had Plaid not been forced upon me, I would never have used it willingly.

[immibis]

You think digital sovereignty is when you are not allowed to do what you like with your account, but must follow someone else's terms and conditions?

[soulofmischief]

It's a complex topic which requires the balancing of some things that may seem at odds.

Yes, digital sovereignty means owning your data and the means to transfer and activate it.

It also covers things like not having to relinquish a personal key or passphrase in order to do so, as that severely diminishes your personal security, erodes privacy and trust, and enables a future society where corporate participation is mandatory and the dissolution of security of privacy boundaries considered essential and unavoidable.

Such a system is horribly anti-consumer, even if it seems nice while the lollipop is still in your mouth.

[Etheryte]

The problem is resource consumption. On some of my servers, scrapers, bots etc make up a vast majority of both the bandwidth and CPU usage when left unchecked. If I didn't block them, I would need to pay for a beefier server. All the while this doesn't give me or my regular visitors any benefit, it's just large corporations driving up my hosting costs.

[diggan]

> The problem is resource consumption. On some of my servers, scrapers, bots etc make up a vast majority of both the bandwidth and CPU usage when left unchecked

What are they downloading, like heavy videos and stuff? Initiating heavy processes or similar?

[radlad]

It takes 125,000 4MB requests to use up 0.5 TB bandwidth, which is the lowest offered by Vultr. I could see this being an issue for personal sites that include photos.

[hansonkd]

> It seems like suddenly half the internet forgot about the term "user-agent", which up until recently was almost always our browsers, but sometimes feed readers, which was acceptable it seems.

Was it really "suddenly"? it seems like for the past decade there has been an ongoing push to make everyone use "chromium" based browsers. I remember 10-15 years ago you would get blocked for not using IE or whatever, even though the site worked fine and there was no technical reason for the block.

It was over 12 years ago when google effectively killed RSS to prevent alternative methods of access.

[diggan]

> I remember 10-15 years ago you would get blocked for not using IE or whatever, even though the site worked fine and there was no technical reason for the block

Reminds me of when I discovered that Google Inbox worked in Firefox, even though Google decided to only allow Chrome to access it:

https://news.ycombinator.com/item?id=8606879 - "Why Is Google Blocking Inbox on Firefox?" - 213 points | Nov 14, 2014 | 208 comments

(correct link to the gist is https://gist.github.com/victorb/1d0f4ee6dc5ec0d6646e today)

I think that "ongoing" push you're talking about was/is accidental, because a lot of people use Chrome. What I'm seeing now seems to be intentional, because people disagree with the ethics/morals surrounding AI, or seeing a large impact on their servers because of resource consumption, so more philosophical and/or practical, rather than accidental.

But who knows, I won't claim to have exact insights into exactly what caused "Chrome is the new IE", could be it was very intentional and they never stopped.

[SoftTalker]

I don't remember Google (search, at least) ever not working in any browser I tried, and I used some oddball browsers over the years. Maybe apps like gmail and docs, but they simply would not work in other browsers. Remember in its early years Chrome was a darling because it was supporting the "modern" web. That was the whole stated reason Google developed Chrome: to support modern, rich web applications, and force other browsers to do the same if they wanted to stay relevant. Nobody guessed that Chrome would eventually be the new IE.

[int_19h]

Google has a very long history of downgrading or even outright blocking off their services based on User-Agent. Them doing it to Opera in particular was a big story back in those "early days". And no, it's not because it wouldn't work - people literally made it work by spoofing the UA.

[Klonoar]

…10-15 years ago?

Try like 20(~+)

[reverendsteveii]

I came here to say exactly this. Of course bots can do things that are shitty but we need to detect and ban the behavior and resist the urge to also try to detect and ban the type of user that tends to exhibit that behavior. If a bot comes onto my page, hands off the captcha to a human, then does human things at human speed and human scale I shouldn't care that they're a bot and I should allow them to do what they're doing. If a human comes onto my page and starts trying to brute force directory structures, mass-downloading tons of huge files and otherwise causing a problem I shouldn't care that they're a human and I should block them. This whole idea of bot detection and blocking seems to be an inversion of what I think is the best design principle we've discovered in the history of software development: build things that do simple, useful things without regard to who is using them or for what, then let consumers surprise you with what they do with it. Banning non-abusive agents is just locking out potential upsides for your app.

OTOH if you make your living serving ads a bot bypassing your monetization is a problem for you. Either you detect and block them or eventually the value of an ad impression in your app will approach zero. So in some cases I guess merely not being a human is the abusive behavior.

[cess11]

As an almost fanatic VPN and Tor user I'm already used to being blocked and circumventing it, usually by exiting through a data center IPv4 but I sometimes check whether they let SeleniumBase access with a chromedriver and surprisingly often they do.

The Internet is a rather hostile place, I don't think that'll change anytime soon.

[rkagerer]

Many (most?) commercial websites have terms of service that ban you from using bots, scripting, etc. It's a similar travesty.

[golergka]

I'm building a service which needs to extract rss feeds from pages (hntorss.com if you're interested). Nothing else. From any rational point of view, website owner would actively want this parser to work as easily as possible — the whole point is for users to see the content you publish!

Alas, I still get rate-limited, 400-ed and others because of user agent and other bot-detection mechanisms.

[chii]

> the whole point is for users to see the content you publish!

no, the whole point (for most sites) is to make money off the users visiting said site (currently via advertising).

Another third party service which slurps the data, and redirect the users to a different site to consume the data means the original site lost the revenue, but paid the bandwidth cost.

So it's understandable that many sites want to block such agents.

[strogonoff]

Even if it is not for profit (or especially so), the point of any publication is not just to get people to know something—you at least want people to read what you wrote and appreciate you for this.

Using Web normally, with search and all, is well-behaved in this regard, but using attribution-stripping technology isn’t.

If your readers don’t know you exist and you don’t know who your readers are or if they even exist, you basically become a ghost writer, content producer for LLMs (and in many cases some commercial LLM operator also makes money off your work, too).

[golergka]

Then you wouldn't have RSS feeds in the first place. I'm talking about sites that decide to have them for one reason or another.

[immibis]

Because they make their money from showing ads to human eyeballs and not to AIs.

[1shooner]

>now we have a new user-agent available, "AI Agents", that somehow is unacceptable and should be blocked?

Giving deference or even exclusive access to certain service clients is as old as the commercial web. The article specifically cites security or other risk as the reason. Of course commercial media on the web today put conditions on the consumption of what they publish: ad-blocker nag screens, paywalls, etc. Usually that's just a commercial interest, but what about other conditions, like a disclaimer for medical or legal advice? AI Agents will cite your content without necessarily the context or due diligence you may be legally or ethically obligated to provide with that content.

Generally, I agree that it holds us back from what the 'Agent Experience' web will inevitably need to become, but there are valid reasons for the incumbent patterns that should be resolved in a mutually beneficial way.

[digitaltrees]

thank you!!!! This is the right answer.

[zb3]

They only care about ad revenue. I guess if bots were paying them they wouldn't need to detect humans anymore..