[Ian_Kerins]

One of the main ideas, we explored here is how scraping has shifted from being mainly a technical challenge to an economic one:

- Infrastructure and proxies have gotten cheaper, but anti-bot defenses have evolved fast.

- Because of that, the real cost of scraping is now the cost per successful result, and spikes of 5x–20x can happen when defenses tighten.

- The bottleneck today isn’t just “can you scrape it?”, it’s whether you can do it profitably and efficiently.

I’d love to hear how folks here are dealing with rising scraping costs or what strategies have worked when data value doesn’t obviously outweigh defense costs.

[joe_91]

Nice concept. I've definitely seen this play out in practice.

A lot of sites aren't impossible to scrape, but they're steadily getting more expensive. We're having to lean more on residential proxies, headless browsers etc just to get the same data that used to be straightforward...