|
|
|
|
|
by dilatedmind
1860 days ago
|
|
would have been useful for something we ran into at a previous job. For legacy reasons large csv files had been zipped instead of gzipped (or some other streamable format), and were causing some process in our data pipeline to consume too much memory. I wound up coming up with a quick solution that appeared to work. Taking advantage of being able to get s3 objects at arbitrary ranges, I implemented seek and read on top. |
|