Hacker News new | ask | show | jobs
by klhugo 2079 days ago
Here is the thing made me a happier person this year: I realized when CSVs take too long to load, I should not be using them in the first place!

I love SQLite now <3
1 comments
karbarcca 2079 days ago
It's always a transcendental experience to move data from CSV into any more structured/flexible format!
link
klhugo 2079 days ago
And fast! I got easily 3 orders of magnitude speedup. I used to load CSVs with pandas and do typical filtering operations... oh boy, some of the stuff would take a hole morning, not to mention the memory usage. I do all that in minutes now.
link
benhurmarcel 2078 days ago
Sorry if that sounds stupid, but then when you get a csv file you write it to an SQLite file before doing computations?
link