[joatmon-snoo]

An example I ran into recently: I wanted to scrape pricing data for used cars, to better inform a friend's decision about what to purchase.

I know there's a relationship between mileage and depreciation, but wanted to have a better sense of what that relationship is to know whether a given car was over or underpriced.

Similarly, if I was pulling that data to build a service of my own to offer to users... is that unethical?

[sroussey]

All of these questions are easily answered by the question: can I run the bot on the same PC I use regularly? If so, then do it there. If not, then don’t do it at all.

[pocksuppet]

This is often really good for your bot, because anti-bot providers are loathe to block what, as far as they know, is a residential CGNAT address. Sometimes you get more success scraping from home with Firefox or Chrome, than with an army of proxy networks.

[fc417fc802]

Why should technical capability to evade countermeasures dictate whether or not something is ethical? My view is that scraping remains ethical as long as your actions aren't causing technical problems for the operator. If anything, a retailer attempting to hide pricing data is what's unethical in my view.

[adolph]

> scrape pricing data for used cars

Time was you could get lovely json feeds from every site by iterating the inspector curl statement. Now-a-days you can't even use Selenium without Cloudflare getting grouchy. Last fall had to make my spreadsheet like a cave-person control c, control v. It wouldn't be so bad if the dealer aggregators' coverage was xor, but you have to dedupe listings. Then there is the whole online salespeople who don't show up at the dealership.

[inigyou]

There's a JavaScript property called navigator.webdriver that returns true if selenium is in use. Obviously, every antibot system checks it. Obviously, you can patch it to always say false.