Hacker News new | ask | show | jobs
by mdaniel 565 days ago
"like this" meaning you want wire PG compatibility, so you'd do something like this?

  UPDATE kv SET value = 'alpha' WHERE key = 'beta.charlie';
  UPDATE s3 SET value = $b64$good luck$b64$ WHERE key = '/some/s3/path';
  LISTEN whatever;
1 comments
PowerfulWizard 565 days ago
I don't know what I want exactly but I'm thinking along the lines that SQL is already doing a lot so it would make the most sense to start with a database interface and augment from there, to try to build a system to handle all the common forms of durable storage used by applications.

The type of situation I'm thinking about is for example storing a blob in S3, storing metadata and a reference to the blob's path in a database row, sending a message into a queue to trigger some async processing, and updating a cache. It would be nice to be able to do this through a single API or service, and it would be really nice to do all this within some type of transaction abstraction that would allow all operations to pass or fail collectively, really really nice if the whole thing could be pay-as-you and scale horizontally-ish on shared infrastructure without managing nodes or slots or whatever.

I'm not a Postgres user so I don't know how far you can get currently and I should probably look into it in detail. Coordinating blob/ject storage, database, and pubsub operations is a pain point for me presently. I think that overall system design is going to prevent a database-type system from being a good idea for blob storage but I would still like to see someone try to put three systems in a trenchcoat and try to make it work behind one interface.

link
mdaniel 565 days ago
I was just trying to get a sense for where the line of demarcation was in your mind. PG has "foreign data wrappers"[1] that allow one to treat external ... whatever ... as if it were a table or procedure within PG. Just stupid powerful, IMHO. It is FDW-specific whether "transaction" means anything to the foreign system, so that may break your mental model but could still get you very close (e.g. BEGIN; UPDATE s3 SET ...; ROLLBACK; may not do anything sensible)

https://github.com/turbot/steampipe#steampipe-plugins and https://steampipe.io/docs/steampipe_postgres/overview may be relevant, although watch out for Steampipe's license

https://github.com/topics/foreign-data-wrapper and https://github.com/topics/fdw are some other examples

1: https://www.postgresql.org/docs/17/fdwhandler.html (although strictly speaking that page is for _authoring_ FDW, not a tl;dr of the concept)

link