[countessa]

don't know the twitter api very well, so....maybe instead of scaling back and doggedly trying to get the current record set, you simply mark the 1st 1000 as having an error somewhere, proceed to the 2nd 1000....come back to the ones giving you problems later. I know eventually you have to come back and get the broken ones, but if you manage to process 80% of someone's millions of followers, then you can start digging into the other 20% a bit at a time and at least provide some value for your customers in the meantime......

I haven't actually checked out your service because i'm not really a twitter person so maybe you do this already, but could you provide statistics based on the amount of data you've got so far? (i know they will be inaccurate), but you can sort of give them as a "moving target", based on x number of your followers type thing. That way the user gets a little bit a value right away.

[mittermayr]

i also tried this and it helped a bit, for every 100 user details request I pull a random set of 100 ids from different places in a user's follower list to minimize getting stuck. it helped a bit. but main problem are still the time-outs.