Hacker News new | ask | show | jobs
by jastr 3678 days ago
This is pretty cool!

What functionality does this provide that Pandas doesn't?

Can it handle tables too big for memory? For example, could I use it to read 10 gigs from PostgreSQL to a CSV.

1 comments

I would recommend that divide one SELECT query for 10GB data into for-loop that smaller SELECT query (like 1MB-data/query) execution and writing table by using LIMIT and OFFSET clauses. In that way, you don't need a large amount of memory.