[deckar01]

Making a network request from a script is not abuse, consuming excess bandwidth is. Scrapers already spoof browser reputation and cycle IPs while abusing bandwidth. Recently I wanted to convert the results of an Autotrader filter into a table so I could supplement it with data they don’t track (towing capacity). It was two pages of results, but requesting it from a script was aggressively blocked by browser fingerprinting. I had to port it to JS and run it in my browser console manually to get the data out, wasting my time.

[coldbrewed]

CDN security SWE here. I have functionally zero interest in what individual people do in terms of automation. I automated my gym's class signup forum during COVID so I get the validity of the use case. You want to work around the mitigations to do the same? Go with my blessings.

The killer is that everything that works for individuals trying to get through the day and make the web a bit smoother is immediately used by industrial crawlers strip mining the Internet, and you can't block one without blocking the other. A future where the web is only accessible via device attestation is extremely dystopian but so is an Internet where every drop of content goes into an LLM training set.

Things like this is why all of the worthwhile content is going to drain into balkanized spaces, and we're all the poorer for it.

[deckar01]

It’s not that dystopian if you are given the choice to trade your attestation for more valuable service. We have been paying for the web by letting advertisers correlate our behaviors together for a couple decades now.

[coldbrewed]

The concern is that attestation turns into another mechanism to monitor and control people; see how age verification turns into surveillance.

[tadfisher]

Why did you need it from a script? Right click -> Save Page As... is still a thing and works quite well in my experience, even with complex React SPAs.

[dclaw]

It's not just excess bandwidth. It's cpu cycles, poor site coding, database issues. Like I said, not everything is backed by a huge company that has the money/hardware/engineering resources to absorb the load.

[transcriptase]

That’s the point. For every legitimate though ultimately unimportant use case like yours there are a thousand others who would scrape the results 86,400 times a day to either try to sell or mirror and plaster with their own ads.