[portInit]

At this point we're considering it a foundational concept to build around - web content changes, so our best option currently is to make the query as easy as possible to change, and alert when things break.

We have done some preliminary work in some AI or other intelligence for pattern recognition to be able to handle structural changes better, but still have lots of work.

But the expanding and querying concepts also make a lot of sense with APIs, which tend to be a little more stable.

[robbs]

IMO, this is the hardest part of maintaining a web scraper. We had ~100 scripts to scrape ~1000 clients' sites and it was, at minimum, 50 hours a week to keep up with changes.

The second hardest part was 30% of our clients all used the same hosting provider, which would start to fail at 10-20 req/s. We had to throttle the sites by IP, cluster-wide.

[portInit]

This makes sense and I am curious about this. Was there consistency between those 1k client sites or were they all rather different? Mind if I reach out?