[pcowans]

Can you be more specific about how large 'large' is?

Edit: without knowing the specifics, you might like to look at Elasticsearch (http://www.elasticsearch.org/). That'll give you an HTTP/JSON API into your data, so you might be able to do all the UI work you need client side, e.g. with Backbone.js (http://backbonejs.org/) or similar.

[sixQuarks]

the csv files are not that big. a few thousand rows. But there are lots of duplicates rows I need to get rid of

[pcowans]

Exact duplicate rows? If you have access to standard Unix tools, try this on a command line:

cat input_file.csv | sort | uniq > output_file.csv