[bbayer]

I personnally dont like Java because of API complexity and this is why ended up with python. I have implemented many crawlers by using Scrapy framework and I believe it speed up development. We have crawled millions of pages without any problem.

Python is very powerful in terms of string manipulation because it has very good language constructs (like slice syntax) which makes development easy. At the beginning it might be a little bit confusing but once you mastered it you really feel power.

Twisted like frameworks also makes good job at this point. It is well-designed, asynchronus and it suits well for multi-tier network applications.