[manmanic]

This is almost cool, unfortunately the product itself (a network of bots to allow websites to be scraped when they obviously don't want to be) seems a little shifty. For example, put these three exhibits together:

Exhibit 1: The ScrapingBee terms and conditions state "We assume that you use the Website Platform and Services legally and ethically and that you have obtained permission, if necessary, to use it on the targeted websites and/or other data sources." This is even backed up with an indemnity clause in which the user has to cover ScrapingBee for any third-party legal claim arising out of their use of the product.

Reference: https://www.scrapingbee.com/terms-and-conditions/

Exhibit 2: ScrapingBee explicitly advertises a feature allowing you to get Google search results via an API call. These results are presumably generated by scraping Google's search pages:

Reference: https://www.scrapingbee.com/features/google/

Exhibit 3: Google's own documentation explicitly states that automated querying is prohibited, so if you use this advertised ScrapingBee service, you are naturally violating Google's terms, and could be liable to cover ScrapingBee's legal costs if Google decide to come after them.

Reference: https://developers.google.com/search/docs/advanced/guideline...

$1MM in ARR is all well and good, but there's a limit to how large this business can grow without being pursued by the websites whose scraping they are enabling, and in the case of Google, explicitly promoting.

[twox2]

It's good that there are businesses like this testing the legality of scraping. The notion that scraping should be illegal is absurd IMO.

[c0balt]

Forbidding scrapping should be allowed and respected, especially scrapping by businesses.

Scraping by itself shouldn't be illegal per se but not respecting policies and using glorifed bot nets won't help anyone. Scraping costs the hosting person/ company money and/ or resources. For example, the advertised use case of scrapping a job board to create an aggregator for job postings is maybe good from the point of someone searching for a job but a hefty punch to the hosted of the job board that has potentially hefty costs (running the board, moderating, ...) without any gain.

Scrapbee is not helping anyone but themselves and is only challenging the legality of commercial botnetes IMO.

Edit: changed 'public botnet' to 'commercial botnet' in the last sentence

[twox2]

It's a tug of war between "acceptable behavior" and "bad behavior", but in the interest of an open internet, this cat and mouse needs to exist. Service providers negatively impacted by scraping can price in their mitigations.

[karterk]

They don't respect robots.txt or have a way for websites to opt out. These might not be illegal but mighty rude.

[simonswords82]

If I'm making a $1m a year in revenue and the absolute worst you can call me is "rude" - I'm down.