Hacker News new | ask | show | jobs
by arianvanp 1 day ago
The litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ out of the box or not

They do not.

3 comments

I don't know who 'they' is here but that's not the point? I would bet that a decent chunk of scraping happens because there is no API (or other machine-focused interface, like RSS).

"pay to crawl" sounds like the absolutely laziest possible way that a particular site could bolt on an API.

has anyone been using this for success? wondering what kinds of pays they are getting. or are the crawlers just avoiding those sites.
"we don't negotiate with terrorists"