[polote]

For all the people that say this is easy. Try it ! That's not easy at all, I've tried it and spend a few weeks to get similar performance. Receiving thousands of request is not similar to making thousands of requests, you can saturate your network, saturated with latency of random websites, get site that never timeout, parse multi megabytes malformed html, get infinite redirections.

My fastest implementation in python was actually using threads and was much faster than any async variant