[cypherdtraitor]

I have done this several times now. CSV just plain sucks.

1. Rip all CSV data into SQLite or another tabular database

2. Do all data manipulations by shifting information between the database and memory. Ideally you pull entire columns at a time. 95% of your runtime is going to be spent pulling and pushing data, so minimize the number of calls however possible.

3. Export the database to CSV and ship it back to the customer.

If you use a particular language a lot, it is worth it to write a text scanner that uses low level APIs to read large CSV files quickly. I usually pipe a million characters at a time, submit most of them to the database, the duct tape the last few characters to the next million that I pull.