[bluelightning2k]

If we all have at least one cloud, why would we seperate out this one piece and run it in Defer instead?

I don't mean to be unkind it's a legitimate question not sarcasm.

It just seems like a subset of Cloud Functions or a subset of Lambda - without being connected to the rest or any of our existing workflow, monitoring, secrets, etc.

[bayesian_horse]

You already set up that workflow. That can be a major PITA.

Also I don't think this is as much about the "compute" of the background job as much as the nodejs app saying "take care of this for me please, got this? Don't bother me again until I ask for the result".

Endpoint handlers are flaky. They aren't guaranteed to run to completion even through no fault of their own. They should ideally have short and predictable run times, otherwise scaling is much more painful. That's why people separate out certain stuff into those background jobs. And getting that just right, scale it and so on can be quite a bit of work, but yes, something like Cloud Functions or Lambda would serve very similar functions. If you set up all the CI/CD pipelines to make that work.

[lytefm]

> If you set up all the CI/CD pipelines to make that work.

I've been "abusing" GitLab CI pipelines a lot for running periodic background jobs that I'd like to have separated from the "regular" backend and without worrying much about deployment.

It supports Cron Syntax for scheduling, manual re-runs, provides secret management and download of artifacts, alerts me on failure and allows me to easily use the docker images in our registry.

Sure, I'm not storing anything in a database or calling the pipeline via an endpoint. But if that's needed, it should probably be "part of the regular backend".

[wittydeveloper]

We primarily focus on helping Node.js developers to build products without investing time in infrastructure while keeping it configurable.

As @thdxr mentioned, our goal is also to provide top-notch queueing/scheduling features that are complicated to achieve with Lambda/SQS/Cloud Functions, such as dead letter queue support, throttling, or a product-oriented dashboard. In the same idea, we plan to provide better integration, for example, as you mentioned on secrets management by integrating with Doppler, AWS Secrets Manager, and more - the same goes with linked deployment pipelines.

[thdxr]

think we see a lot of unbundling happening right now (I work on https://sst.dev/ so see a lot of different setups)

a lot of times these specialized services can really nail a more narrowly scoped feature set to the point where it's worth having stuff running outside of your primary cloud

it always depends, for me personally that bar is very high because like you said, now you have to copy paste secrets into yet another service among other things

some services are so good, like planetscale, especially compared to the cloud option that I'm willing to do that