| The simple case is simple, but anything beyond that can get challenging. Some examples: Multi-step functions. This requires chained jobs, and your dependencies get messy. Operating this kind of sucks. Change management here is hard. It's much easier and faster to write a single function using Inngest; basically any engineer can do it without learning infrastructure. Absence of events/activity (eg. when a user adds a product to the cart, send an email if they don't check out). You need to start writing crons and depending on implicit state that occurs within your DB. I've done this; it becomes really buggy. Much nicer to do `step.waitForEvent` in procedural code. Fan-out (eg. when a user signs up, add them to Stripe, intercom, etc, or mass ecommerce imports). You can definitely handle this with individual jobs, though this starts to stack up and then monitoring is overblown. Retries and dead-letter queues. It's much easier to replay functions en-mass, or replay events that have failed, without worrying about dead-letter-queues and redrives. Config generally sucks. You can manage concurrency, backpressure, etc. yourself, alongside managing the infra, but that's no fun. Or, you can use some existing queues and then manage cloudformation, terraform, infrastructure code, DLQ config, etc. yourself too. Batching is annoying to build. I'm not sure how you'd batch high-volume events or jobs in Dramatiq. Maybe you can add your own observability, deploy new workers, and try to avoid batching — deal with that yourself? There's more, too. Rate limiting, concurrency controls (eg. per user, not per job), branch deploys/preview environments/staging envs, are all pretty frustrating and require infra investment. As with anything, handling the initial case is pretty easy but quickly gets hard. |
It is hard to scale background tasks when you hit a high level of concurrency, as you've mentioned.
Your press releases makes it sound like the initial setup of background tasks is hard, and doesn't mention the harder stuff.