[mikeocool]

Whether or not scrapping publically available websites is unethical is probably up for debate. In some cases at least, courts have found it to be legal, even when the site is throwing up technical barriers or issues cease and desists.

What is likely unethical is the fact that they offer residential proxies. The residential providers of those proxies are frequently not aware they’ve been opted in to provide such a service.

[eab-]

> courts have found it to be legal

≠ ethical

[OhSoHumble]

I don't know if this falls under good ethical behavior but the one time I needed to mass scrape sites was to build an affordable housing directory for the Bay Area. Whether or not a unit of housing was under rent control was only available through private entities or apartment hosting sites and it was scattered and hard to find.

In my opinion, a directory of subsidized housing should have been provided by local governments and not through a plethora of real estate websites.

[inigyou]

Some are aware. I know some providers that pay you to run a proxy, but the pay isn't very much.