Hacker News new | ask | show | jobs
by new_test 4278 days ago
>loading those files into Pandas or R

What is "massive" for you? I was under impression you can't use R or pandas for anything that doesn't fit into memory.

1 comments

As for massive - something like daily options data for 3000 stocks, spanning a number of years, with information down to the tranche level (let's say 60 million rows if stored in a relational database fashion). In my case the analysis can be done on the stock level though, which means that only a 3000th of the dataset needs to be loaded into memory at any time.