[NTH]

My main problem is that it has awful support for functional programming, which I find to be a really helpful way of doing something like a web crawler, where you're essentially describing a computation to parse some input. I would use F#, because it offers powerful functional programming tools, is on .NET / VS 2012 (not sure if that's a pro or con for you), and has type inference (so you get the benefits of static typing without the cost of writing out the types of everything).

You should probably check existing web crawler solutions to see if you can adapt them before rolling your own.