Hacker News new | ask | show | jobs
by hmpc 2102 days ago
Don't you need your read to be line-oriented to avoid breaking up lines?
1 comments

You might want to read from the source file line oriented until you find the start position of the first line to copy. Then copy from the end as described in large chunks (I would do 16Mb chunks assuming ram is sufficient).

The sparse file suggestion a sibling made is really great. Otherwise, you could reverse the chunks as you write them, and end up with a fully reversed file, or just have to be careful about block boundaries and know that contents are forward, but blocks are backward. I wouldn't want to bulk read a 700g file with line processing, if it's possible to ignore that.