Hacker News new | ask | show | jobs
by maxlybbert 2605 days ago
It’s easy enough to read a file in chunks, parsing out the information as you go. This limits memory use as long as you release the chunks when you no longer need them. The operating system can swap out memory as-needed, even if you didn’t get the memory from mmap, so it’s irrelevant where you store the parsed data.

Unless you actually need to read the file multiple times (compared to looking at the parsed in-memory data multiple times), this should be fast enough.