Hacker News new | ask | show | jobs
by faangguyindia 83 days ago
Most problems with overgrowing data can be fixed by having data deletion rule.

Many people just keep adding data and think "maybe it will be useful in future" till their system goes down.

Many of your data is essentially useless for anything in future.

You can simply have data retention policy and for most app this ensures your data does not grow top huge

2 comments

In some cases you have no choice but to retain the data, e.g. due to compliance. But the good thing is it doesn't have to be in Postgres. You can periodically offload data to a lakehouse, then delete it from Postgres. If the table is partitioned, delete should be cheap.

I'm guessing with Neon, since their storage is a lakehouse, you get this for free.

This is orthogonal to many classes of write throughput. And in fact in many cases deletes themselves are effectively adding to the write workload