[cynwoody]

HtmlUnit† is also effective in such cases. HtmlUnit is intended to automate testing of websites. However, the very facilities that enable it to be useful for that purpose also make it useful for scraping.

A few years ago, I wanted to analyze retail store customer feedback data collected by a third party company. The stores were franchises, and the third party was anointed by the franchising company. The data was presented to the user (franchisee store management) via a fancy web site with its own opinion about how the data should be analyzed. My opinion differed. I wanted the data in low-level, RDBMS-friendly form, so that I could recast it every which way (and come back and do it again a new way I thought of). However, such was not forthcoming (big company, little franchisee).

The solution was to make a robot that put the third party company's portal through its paces at the finest granularity, scraping the numbers into a DB as they tediously appeared. The robot was in JRuby††, allowing access to HtmlUnit's functionality without the tedium of Java coding. It was slow, but I didn't care — run it overnight once a month, then run reports off the DB generated.

The coding approach was simple: Pretend you are a user. Access each page, starting with the login page, and do what the user would do. Scrape the interesting numbers as they appear. Append appropriate rows to the DB.

†http://htmlunit.sourceforge.net/

††http://jruby.org/