[psychometry]

Netflix was my first introduction to webscraping back around 2004 (with Perl and WWW::Mechanize) because they didn't see the need for an API. They've never done anything more than pay lip service to it since they finally caved in and made one. Guess it's back to doing that, or using an alternative like TheMovieDB [1], which has a nice API.

[1] http://www.themoviedb.org/

[burntsushi]

You might be interested in a tool I wrote called goim[1], which is a command + library (written in Go) that loads all of IMDb into either a PostgreSQL or SQLite database. (It takes about 12 minutes for the download and loading of the database from scratch on my personal machine.) It supports complex queries (with fuzzy matching) really easily on the command line. For example, this will find the top 5 ranked Simpsons episodes with "homer" in the title:

    goim search '%homer%' {show:simpsons} {sort:rank desc} {limit:5} {votes:100-}

And if you're like me and rip episodes off your Simpsons DVDs that are named "S04E12.mkv", then you can rename all of them in one swoop:

    goim rename -tv 'the simpsons' *.mkv

It will automatically find the `S04E12` in the file names to get the correspondence between episodes.

Of course, the database has more than Simpsons episodes in it (and renaming works with movies too), but it's my go-to example. :-)

Want to try it out? It's easy. Make sure you have Go installed, then install and load and search:

    go get github.com/BurntSushi/goim # ~ 30 seconds (compiles SQLite)
    goim load -db goim.sqlite # ~ 1 minute, only movies/tv shows
    goim search -db goim.sqlite '%homer%' {show:the simpsons}

Loading/searching is a bit slower with SQLite than with PostgreSQL, but it's more convenient. (e.g., The search command above is over 12x faster with PostgreSQL on my system.)

[1] - https://github.com/BurntSushi/goim