Gonna zag here. If you take a step back, cloudflare has been paving the path for pay for crawl. I think it's a noble and ambitious goal.
While I can understand why you would be alarmed, I can point to almost two decades of lamenting on this forum about how we need better ways of rewarding content creators than ads. Well, this is it.
Moreover, these products weren't built in a vacuum. Most threads about Anthropic and OpenAI have complaints about how these companies were built on stolen content. There's a reality we have to face here and we can't have it both ways.
Is it a better way though? The problem with ads has always been their abusive nature. From unrestrained pop ups and clickjacking in the 90s to today's pervasive surveillance and profiling.
Cloudflare turnstile is a pop up. This product is pervasive surveillance. It having a cf logo doesn't change that or ameliorate the many many abuses that two decades have shown are part and parcel.
What's incredible is that you have businesses paying CloudFlare to stop their content being ingested by AIs (OpenAI, Anthropic, Self-operated scrapers).
And at the same time, they're paying SEO experts to make that same content easier to be ingested by systems (Google and other Search Engines) which use it for their own AI offerings.
Are you going to be able to make your online content available to Google Search but unavailable for Google Gemini?
You said "a similar range of services". Basic DDoS protection is probably important, and there are other companies doing that. Beyond that, it's less obvious.
They are advantageous to leverage in certain situations but essential they are not. We're used to, in the technology industry, looking for or creating problems to solve with services we are aware of. Moving back to necessity and need, do we really? Are we being objective? Most of the time, no.
The complaint is that the offer is a great deal with no downsides for consumers, and this is likely to result in Cloudflare having a lot of power (which they currently don't have) as a market maker. This position as market maker would grant them the power to extract economic rent from the web economy by charging both sides of the web provider and web consumer market to get access to the other.
You don't need to speculate about whether a tech company with a monopoly over a major distribution channel will extract rent from it - it's incredibly profitable to do so.
It's really telling that they're starting to block anything automated ever that hasn't gone through their ID verification process. It's like literally every company that exists in the world is taking advantage of the dystopia at once
no, they're giving tools to their customers who can choose freely to block or not block bots. Without those tools, the people who run sites and offer content are just flying blind. I struggle to see how this is a bad thing in any way
My issue is with them defaulting every site to blocking any bot that hasn't gone through their verification process. It's not about the existence of the setting, but rather the behavior to enforce it by default.
For any one of their product there is a good opportunity to build an open source alternative or something like it! Can be hard to work around they have the benefit of being able to have negative unit economics on lots of infra products... But people succesfully built tons of alternatives to google analytics and similar.
Open source for bot protection specifically would be difficult. If I as a bot developer can see the tests you run I can just modify my bot to pass them (either trivially or by brute force).
While I can understand why you would be alarmed, I can point to almost two decades of lamenting on this forum about how we need better ways of rewarding content creators than ads. Well, this is it.
Moreover, these products weren't built in a vacuum. Most threads about Anthropic and OpenAI have complaints about how these companies were built on stolen content. There's a reality we have to face here and we can't have it both ways.