Hacker News new | ask | show | jobs
by dymk 790 days ago
38TB of large objects stored in Postgres right here
1 comments

A hero appears!

The client I use currently, npgsql, supports proper streaming so I've created a FS->BLOB->PG storage abstraction. Streamy, dreamy goodness. McStreamy.

I am confused and curious. Can you please elaborate on how streaming is related to pg storage?
If you can't stream data the user is sending through the server to the database you have to buffer the entire payload in memory. This doesn't scale well but can work for smaller payloads.

You CAN stream bytea but:

* Good luck finding a client that supports it(npgsql does actually)

* It doesn't support support start the read/write at any location other than the start. So you'd have to basically build the Lob(Large Object) functionality yourself on top to support upload resume and other use cases.

With Lob even if your client doesn't support streaming, you can mimick it by writing chunks into the Lob.

What kind of streaming from user are we talking about? User interaction, such as mouse move, clicks?

Does each interaction essentially create a SQL update statement? And are you trying to avoid buffering those statements?