Hacker News new | ask | show | jobs
by yummypaint 2419 days ago
Do you have a sense of how the performance would compare to chunking? My naive expectation is that loading data from disk to swapped memory involves writing that data to disk (even if it will only be read).
1 comments

Chunking can speed up processing even if the dataset fits into memory because you are interleaving disk reads and computation and the OS is likely to prefetch the next chunk into the read cache while you're still busy computing.

On other problems chunking doesn't work at all and just mmaping or dedicating giant amounts of swap are better strategies. It depends on the problem at hand