|
|
|
|
|
by srean
2860 days ago
|
|
> in which case there aren't too many references until the subprocesses get to work. In my workload that's exactly when it hits. We ran into this when sharing different parts of a huge matrix with different workers. We had to be extra careful that we did not create new references in the subprocesses. We were operating at scale where if we got it wrong OOM will kill us. Working with memory mapped arrays are more forgiving. |
|