[timkly]

I've done a little bit of domaining myself and the first thing to take away from any list is that recently dropped doesn't always equal available. Also you have guys that are running massive lists that perform thousands of buys requests per second across a massive number of domain name sellers in order to ensure they get to purchase the domain as soon as its released.

In terms of obtaining information, most whois queries can be performed via command line utilities... so to start you off here is a good list for whois servers (http://code.google.com/p/whois-servers-list/). Finally, check out each service, some will allow queries which will return true or false to being registered and generally you get a lot more of these requests then complete lookups (without being IP blacklisted)

Finally, in terms of building and managing an index, I believe manual crawling is the only option available... and start with dictionary terms and work out.

Edit: Read this as well - http://www.dotweekly.com/pending-delete-domain-name-drop-lis...