Hacker News new | ask | show | jobs
by amelius 2077 days ago
I can't attest to the usefulness of pytorch's multiprocessing module, but using python's multiprocessing module feels like low-level programming (serializing, packing and unpacking data-structures, etc. where you'd hope the environment would handle it for you).
1 comments

I found python multiprocessing to work well to parallelize deep learning data loading and preprocessing, because all I needed to communicate was a couple of tensors which are easy to allocate in shared memory. I didn't need complex data structures or synchronization.