Hacker News new | ask | show | jobs
by HackerThemAll 21 days ago
Using server-side cursors is a sign of bad design. You use the PostgreSQL server's resources as a cache when you're fetching and processing stuff row by row, which is very bad. Just get the data you requested in the first place in one go and do your stuff in your app. Or process all data on the server and then get the processed data in one batch as well.
2 comments

As said above, SQL is a set based language and one shouldn’t write a cursor to deal with individual rows unless no other way.

And if writing a server side cursor, probably better to write a stored procedure /function and put the cursor and its logic in it, and then call that rather than handle in application

two different things are getting called "cursor" here. queryset.iterator() isn't server-side row-by-row processing, it's one set-based query streamed to the client in chunks so a big export doesn't materialize in memory all at once, "get it in one go" is exactly the OOM it avoids.

the honest knock on the streaming kind isn't "bad design", it's that it holds a portal open server-side, which is why it can't survive transaction pooling. so the pooler-friendly fix isn't one giant fetch, it's keyset pagination (where id > last limit n), stateless and constant memory, which is what we moved those paths to.