Does Fly let you run multiple commands in separate Docker images? That's usually the pattern on how to run a web app + worker with Docker, as opposed to creating an init system in Docker and running (2) processes in 1 container (this goes against best practices). The Fly docs only mention the approach of using an init system inside of your image and also tries to talk you into running a separate VM[0] to keep your web app + worker isolated.
In either case I think the price still doubles because both your web app and worker need memory for a bunch of common set ups like Rails + Sidekiq, Flask / Django + Celery, etc..
It's interesting that their bash init uses fg %1. That may return only on the first process changing state, rather than either process exiting. It should probably use this instead:
It sounds like you're asking if we offer some alternative between running multiple processes in a VM, and running multiple VMs for multiple processes. What's the third option you're looking for? Are you asking if you can run Docker inside a VM, and parcel that single VM out that way? You've got root in a full-fledged Linux VM, so you can do that.
> Are you asking if you can run Docker inside a VM, and parcel that single VM out that way? You've got root in a full-fledged Linux VM, so you can do that.
On a single server VPS I'd use Docker Compose and up the project to run multiple containers.
On a multi-server set up I'd use Kubernetes and set up a deployment for each long running container.
On Heroku I'd use a Procfile to spin up web / workers as needed.
The Fly docs say if you have 1 Docker image you need to run an init system in the Docker image and manage that in your image, it also suggests not using 2 processes in 1 VM and recommends spinning up 1 VM per process.
I suppose I was looking for an easy solution to run multiple processes in 1 VM (in this case multiple Docker containers). The other 3 solutions are IMO easy because once you learn how they work you depend on the happy path of those tools using the built in mechanisms they support. In the Fly case, not even the docs cover how to do it other than rolling your own init system in Docker.
If you have root, can I run docker-compose up in a Fly VM? Will it respect things like graceful timeouts out of the box? Does it support everything Docker Compose supports in the context of that single VM?
The document you cited (I wrote it!) is entirely about the different ways to run multiple processes in 1 VM.
There's no reason I can see why you couldn't run a VM that itself ran Docker, and have docker-compose run at startup. I wouldn't recommend it? It's kind of a lot of mechanism for a simple problem. I'd just use a process supervisor instead. But you could do it, and maybe I'm wrong and docker-compose is good for this.
What you can't do is use docker-compose to boot up a bunch of different containers in different VMs on Fly.io.
Not sure how Ruby works, but can you not run the workers and the web server in the same process? In our Node.js apps, this is as simple as importing a function and calling it.
Most of the popular background workers in Ruby run as a separate process (Sidekiq, Resque, GoodJob). The same goes for using Celery with Python. I'm not sure about PHP but Laravel's docs mention running a separate command for the worker so I'm guessing that's also a 2nd process.
It's common to separate them due to either language limitations or to let you individually scale your workers vs your web apps since in a lot of cases you might be doing a lot of computationally intensive work in the workers and need more of them vs your web apps. Not just more in number of replicas but potentially a different class of compute resources too. Your wep apps might be humming along with a consistent memory / CPU usage but your workers might need double or triple the memory and better cpus.
Yeah, it definitely makes sense to be able to scale workers and web processes separately. It just so happens that they app I work on for my day job is:
1. Fairly low traffic (requests per minute not requests per second except very occasional bursts)
2. Has somewhat prematurely been split into 6 microservices (used to be 10, but I've managed to rein that back a bit!). Which means despite running on the smallest instances available we are rather over-provisioned. We could likely move up one instances size and run absolutely everything on the one machine rather than having 12 separate instances!
3. Is for the most part only really using queue-tasks to keep request latency low.
Probably what would make most sense for us is to merge back in to a monolith, but continue to run web and worker processes separately I guess. But in general, I there is maybe a niche for running both together for apps with very small resource requirements.
In either case I think the price still doubles because both your web app and worker need memory for a bunch of common set ups like Rails + Sidekiq, Flask / Django + Celery, etc..
[0]: https://fly.io/docs/app-guides/multiple-processes/