[jonesetc]

I'm not preaching the elixir word here either. I've never used it for anything real and just dabbled a bit as a toy. I'm just trying to help get this conversation going because I'm also interested. My experience is in java, node, and python primarily which covers a good bit of ground as far as the different concurrency approaches.

Anyhow, your characterization there I don't think is quite right. For one Java does not typically use a background job set up, because it is a first-class multithreaded runtime so it seems more likely prepared to use a threadpool for some concurrent processing before it say tosses things out to a task queue. Similarly as most java web servers are on a thread pool model for concurrency as opposed to async IO, you'll also have true in request concurrency for things like making a few calls to the DB and then collecting them. This thread pool is not well abstracted typically, and the mutable object model leads you to writing classically tricky multithreaded code with data races and the like.

For something like node where you are not really likely to be using threads, you are pretty hard tied to async IO as your method of concurrency. So you have a really easy job if all you're doing is async calls to say a DB with no need for data processing. Here once you need real computation you will have essentially no option other than using a task queue.

Then when it comes to python you have essentially the worst of all worlds. it's not easy to do concurrent IO or computation without some form of background processing.

I believe that when it comes to the elixir runtime the idea is that they have removed async IO from the mix and instead put up a concurrency API that is as simple to manage as async IO in something like node. Under the covers is all delegating out simple blocking code to a thread/process pool, but the immutable data model and API makes it so that data is not shared across the pool, trying to keep the classic concurrent mistakes hard to make.

Nothing that can be done in elixir seem to be impossible or even astoundingly hard in some other technology, but the unified API for all forms of concurrency seem like a nice way to make it a runtime that can handle many different types of workloads in a simple and cohesive way. That simple way also seeming to not need you to bring in things like a redis task queue where you generally would need to use it or some other external solution.

[joelbluminator]

"Java does not typically use a background job set up, because it is a first-class multithreaded runtime " : Is there really no equivalent of Sidekiq in java world, or that's just rarely used? I would assume a separate process would need to run over the jobs (so can be scaled easily) so then you need to write the jobs to some shared memory like redis. Is this rarely used? https://docs.spring.io/spring-integration/reference/html/red...

[lfn716]

The advantage of Java and Erlang the author was referring to is that you don't need extra technology. With sidekiq, you must setup Redis and often separate instances to process the job queues. With Node and websockets in Ruby/Python, you typically must move any CPU task to jobs/task queues.

Sure, you may want to move your Java/Erlang jobs to Redis/SQS/etc for different reasons, but it is not your starting point. And given you were explicitly talking about CRUD apps, being able to have everything you need with only your app + database, without a need for Redis, workers, or tasks queues, leads to operational simplicity.

[joelbluminator]

But part of the idea behind a queue job is not to overwhelm the server with heavy jobs that aren't urgent. Even with a thread pool, in java, you may still overwhelm the server with heavy jobs like sending emails, if you do not clog the threads you may still spike the memory - do you really want your server and your background jobs to share the same memory and resources?. So the separate instance make sense to me - they are a feature in fact. I really want to know from someone from the java ecosystem more about how they handle heavy background jobs. I get that Elixir may have this part done in a nicer way, but still ,Sidekiq is a VERY polished tool.

[cpursley]

> do you really want your server and your background jobs to share the same memory and resources

Erlang/Elixir processes don't share memory (Actor model) like Java threads do, giving it a distinct advantage (see Akka for Actor model in Java).

But yeah, for heavy stuff even in Elixir, you might want to outsource that to a Job server (which there are many libraries in Elixir for this).

[joelbluminator]

I'm really asking because I don't know - in java world background queues are actually rare?

[jonesetc]

probably a bit of loose wording there. It's probably not what either of us would call objectively rare, but it is comparatively rare compared to many other runtimes. There's quite amount of this that can be avoided with some vertical or horizontal scaling depending on what you are doing with these events.

For instance you linked the spring application events integration with redis, while that is definitely an option it's not one you'd immediately reach for. Instead you'd more likely set up and use application events and/or domain events (if using spring data) handled locally with thread pool. The benefit here is that of course way less set up and complexity, but you can also handle these events sync or async.

[joelbluminator]

"This thread pool is not well abstracted typically, and the mutable object model leads you to writing classically tricky multithreaded code with data races and the like" : If each request gets its own thread I don't really see the problem (they will not override each other's shared objects). Yes you sometimes run into stale reads/writes which you can solve with db locks etc.

[joelbluminator]

btw Sidekiq also works by using a thread pool as far as I know