[elcritch]

Thanks for putting this out there!

There are some projects out there using memory mapped files to do fast CSV parsing. Could be a nice way to speed up the memory loading and scroll it in real time. Can't find the link to the library I saw it used in, but it might be an interesting venue to consider. Another library that does it seems to be astropy fast ascii IO module [1].

[1]: http://docs.astropy.org/en/stable/io/ascii/fast_ascii_io.htm...

[camkego]

Try benchmarking OS read() calls vs. either sequential or random reads using memory mapping, whenever I do this OS read() calls end up being quite a bit faster.