Hacker News new | ask | show | jobs
by joshribakoff 3222 days ago
What's up with the built in queue? I feel like that belongs in a different script. For one, the built in nodeJS queue is useless in a multiple server environment. You'd still need a distributed queue since this built in one is only local to one server/thread. So the built in queue becomes redundant/pointless for any kind of solution that needs to scale
2 comments

I was first wondering about all the complexity in the API as well (why a built-in queue, webhook, retry policy and storage interface when the actual transaction I'm interested in is just "url -> pdf blob"?)

However, I think this is necessary if you want to fit it into a microservice with a REST interface. For REST, I think the usual expectation is that a) the request returns quickly and b) you can submit any number of requests in parallel. Given that loading a page into headless chrome, rendering it and generating a pdf is both resource intensive and time consuming, I guess you need some way to decouple that process from the interface.

Not the OP, but maybe he wanted to keep the install as simple as possible without requiring something like redis. I also haven't looked at the code, but a queue per process is still useful as long as the results can be accessed from any process. Not sure if this is the case. If not, you would seem to be right, in that subsequent requests after "queuing" the job could go to another process/server not aware of the queued job.