[bilekas]

Am I understanding this correct in that you can basically automate monetizing your web/api content to everyone or just agents ? Because I would be very much in support of charging agents per request, but I would want to still offer humans a free experience.

[Faaak]

Depends on the website though. I want LLMs to scrap my B2B website, because then it's shown to the user and they will likely use my product afterwards

[wpapper]

I’m a PM on the team that is building this. We want to offer a range of options, from charging everyone to charging unverified bots to simply charging users who exceed rate limits. We don’t want to add a dependency on a particular detection mechanism, but we do want to offer a variety of choices depending on how people want to filter.

Feel free to email me at (my username)@(my company) with feature requests or feedback!

[ygouzerh]

Their example of an /api/premium is quite nice! You could you like keep existing pages free, but provide specific output content for llm!

So if: cost monetized API < cost configuring scraper for your website OR feature provided by premium api > data got by scraping, then some people/business will likely pay

[jf93ap29sh]

If not built-in, you can probably put it together through Cloudflare itself.

If a request goes to the protected path, if detected as bot: hard HTTP redirect to the path set in the monetization gateway, if human: allow and don't redirect.

[ryan_n]

Is there actually a reliable way to differentiate human from bot?

[mpeg]

There are reliable ways of differentiating human from cheap, bulk scraping bots.

But if the bot is advanced / expensive enough, it gets a lot harder. Where this product's market sits is in giving a paid way to access content compared to having to spin up bots that run js, from real IP addresses, etc. all of which are more expensive

[xur17]

Agreed. To me this feels like the perfect solution for websites and ai crawlers. Instead of having crawlers paying proxy services and captcha solvers, they can pay the website itself. As a web scraper, I'd happily pay the website provider to get access if it meant easy access to the content. Heck, as a human, I'd pay to avoid the dumb captchas.

[cphoover]

As I understand it as models driving agent behavior of headless browsers are getting more and more sophisticated it's getting harder to reliably predict.

The same way LLM's without watermarking cannot be reliably classified as "not-human" neural-network driven scraping tools are getting harder to detect.

Cloudflare, and DataDome position themselves as companies that can detect automated traffic using things like IP reputation, behavioral signals, timing... But these things can be faked through proxy-networks, human behavior signals can be imitated with generative AI the same way text can be, web bots can utilize neural networks to generate trajectories and timings similar to those of humans.

If you can have an AI use a browser the same way a human can how can you distinguish the two?

[arbol]

If you're on a proxy network then you're detectable by TLS handshake timings, for example. You are right though that agent mouse and touch behaviour is almost inseperable from human data these days. So you need to use a combination of techniques in order to detect automation. At Prosopo we check TLS timings, SIMD performance, page similarity, fingerprint proofs, agent honey pots, JS inconsistencies. Rate limiting is surprisingly effective as an initial deterrant. There is no magic single thing that works for all clients, it's a case of learning distinct bot operators' behaviours.

[carlosjobim]

Unless you have people's biometric data, you won't be able to separate agents from people. Except by payment.

[hedora]

Agents will be able to pay orders of magnitude more than humans, since they can just cache the documents at openai or anthropic, then use them over and over.

[carlosjobim]

But then the cost to access a HTML page will also have to be thousands of dollars, since it can only be sold five times. (Once to Anthropic, once to OpenAI, once to Google, once to Meta and once to Apple).