[sixQuarks]

the csv files are not that big. a few thousand rows. But there are lots of duplicates rows I need to get rid of

[pcowans]

Exact duplicate rows? If you have access to standard Unix tools, try this on a command line:

cat input_file.csv | sort | uniq > output_file.csv