[ricardo81]

Crawling websites behind Cloudflare can also be problematic if they (CF) decide that your bot doesn't fit their definition of OK. This is problematic for new search engine entrants and a multitude of other services, particularly given how many sites now live behind CF.

Years back their DNS service also stopped honouring ns_t_any requests (for reasons of DDOS amplification apparently).

I do tend to agree with you about centralisation, gatekeepers particularly.

[AtNightWeCode]

You can't run around and crawl other peoples sites. That time is long gone.

[ricardo81]

Not sure if you're being sarcastic!

In the end for any scraping they're just raising the barrier of entry. Automated browsers, residential proxies, captcha services just make it more involved for those determined to hit a URL successfully.

Not necessarily a bad thing, but the line and grey area and the definition of a 'legitimate' request varies, and one entity as a middle-man deciding that is less than ideal.

[AtNightWeCode]

Not sure where you coming from but let us go back 10-15 years when there was an open market for commercial crawlers and IP ranges to be used for it. You sold shoes and scraped all other competitors for instance. That era is over.

For legitimate interests today including search engines and services for price comparison that data is often provided for free.

There are design patterns used today that does among other things provide incorrect prices to scrapers.

Scraping is illegal in most western countries btw.

[ricardo81]

>For legitimate interests today including search engines and services for price comparison that data is often provided for free.

Can you explain that further in the context of search engines, new or existing, need to crawl websites and Cloudflare are a barrier to entry? You seem to contradict yourself.

>There are design patterns used today that does among other things provide incorrect prices to scrapers.

If you say so, and hopefully they do it 100% correctly.