[jmuguy]

Not having any sort of basic disk/storage solution has been painful for us. There's been a lot of situations where I would love to store a big blob of data on the disk while processing it but if that Heroku dyno reboots - poof, its gone.

[jrvarela56]

What kind of workflow are you trying to fit?

My impression is that this lack of 'instance storage' is by design, as well as other constraints that can be mapped to these: https://12factor.net/

[jmuguy]

I imagine Heroku would say the same, and thats fine. We deploy a bunch of Rails apps, backed by Postgres, via Heroku and most of the time what we're doing fits neatly into the 12 factor definition.

However what happens if you're doing something that doesn't really fit that definition. For instance we have a Rails app with some background workers doing data processing. I would very much like to have these workers just dump to disk, so I can take a big chunk of data and move it into Postgres at once. But I can't do that with Heroku.

So basically this is something that Heroku isn't designed to do, but its also something I would rather not need to go one level of abstraction deeper, to AWS directly for instance, in order to do. And its also something that every single one of their competitors offers.

[jrvarela56]

My hunch is that if you try to build your solution with the constraints enforced by Heroku your system will be more resilient.

If it's about being easy to dump stuff to disk, using S3 or even Postgres for something like this could become easy after a few attempts. Another pattern that would come out of this exercise is trying to deal with that job in parallel by breaking it up into sub-tasks (many workers at the same time).

Anyway, too little context for me to flesh out a solution but the gist is that in general these constraints nudge you in a 'healthy' direction when designing systems.

[mrkurt]

Boy howdy do I agree with this. Filesystems are immensely handy. We (a sometimes competitor of Heroku) shipped persistent volumes very early and it's been amazingly empowering. The DX gets difficult, but we're ironing that out over time