[yaseer]

So what I'm suggesting is slightly different.

This outputs a series of clicks and keyboard events. It doesn't handle the logic necessary for most JS-heavy web-apps.

Scraping an infinite scroll is a good example. A naive algorithm is - scroll, wait, scrape, scroll, wait, scrape...

We have a slightly more sophisticated algorithm to deal with this. Also, what happens when you encounter a page with multiple iframes?

I'm essentially talking about a library which works with higher-level abstractions than just click and type, perhaps 'ScrapeInfiniteScroll' - and similar operations.