Hacker News new | ask | show | jobs
by myroon5 2977 days ago
It looks like this downloads them one-by-one instead of downloading at the same time.

Is there a simple Python equivalent to Java's .parallelStream().forEach() that would allow these calls to easily be run in parallel?

1 comments

Sure thing...

    from multiprocessing import Pool
    with Pool(8) as p:
        p.map(function, sequence))
Neat.

I see that you explicitly specify 8 as the number of processes here. In Java, parallelStream() will pick a sane default for you if you haven't previously specified (based on the number of available processors, I believe). Is something like that possible in Python?

The ThreadPool class picks a sane default (number of cores), but I believe it uses Python threads instead of processes.
You could just use threads here too - CPython releases the GIL during IO.

  from concurrent.futures import ThreadPoolExecutor
  with ThreadPoolExecutor as executor:
      executor.map(function, sequence)