|
|
|
|
|
|
by skybrian
298 days ago
|
|
|
Apparently there’s a setting for each website to turn pay per crawl on or off, and they also control pricing: > While publishers currently can define a flat price across their entire site, they retain the flexibility to bypass charges for specific crawlers as needed. This is particularly helpful if you want to allow a certain crawler through for free, or if you want to negotiate and execute a content partnership outside the pay per crawl feature. https://blog.cloudflare.com/introducing-pay-per-crawl/ So it’s more like Cloudflare is enabling pay-for-crawl by its customers. There is a centralized implementation, but distributed price setting. This seems more like a market. It’s unclear to me whether Cloudflare gets a cut. |
|
|
Peak giving-Matt—the-headspins would be if JS stepped and made the crawler market for India.