[klibertp]

> * The ease of concurrent programming, e.g. emacs-aio.

There's no such thing. emacs-aio is an extension bolted on top of generators and promises, which is how it was bolted on in Python too - so not bad by itself - but the problem is that nothing in Emacs core supports it. Async in Emacs is still, in 2022, a callback hell, and it's not even supported in newer APIs, like completion-at-point (which is awful - I understand that the origin of this is minibuffer completions and that it might be justified to do everything synchronously, but in-buffer completions should not freeze the editor!)

Now we have threads, but seemingly nobody uses them - not surprising, given that last time I tried I got a segfault pretty quickly (fixed since then). Still, threads? With locks and semaphores? Didn't we all agree that these are not the greatest primitives for concurrency? What about channels or async streams? Not to mention, the threads normally should be preemptively scheduled, while currently they are "mostly cooperative" in Emacs. IOW you can still run code in a thread that will not return control to any other thread, as long as it doesn't do IO. Which would be fine with coroutines, but these are supposedly threads! So you pay in memory for threads but get coroutines, but with pretty fuzzy notion of what's atomic. It's a cosmic horror story.

I'm trying to write a "guide to modern Elisp programming" - there were definitely very interesting and good developments in Emacs Lisp over the last 5 years, and generally the language, coupled with convenience libraries like cl, seq, s, f, and so on, became a very productive environment. EIEIO (and cl-structs) and multimethods (true multimethods, which landed at some point in cl-lib without much fanfare, though they really deserve more attention) are incredibly expressive if you don't mind a bit of syntactic overhead (clojure-style . and .. would be appreciated). cl-loop is incredibly versatile tool that lets you declaratively state almost any kind of iteration and reduction. There's object inspector, there's a package supporting many kinds of refactorings, and of course helpful for rendering information about commands, functions, and variables. It's overall great and productive environment... until you try doing async, unfortunately.

[ywei3410]

Regarding threads, they are definitely more co-operative co-routines than threads.

However, if I recall there are only a few calls which can switch the context; the usual culprits, such as `thread-yield`, `sleep-for`, `accept-process-output` and atomicity is guaranteed apart from when you use these calls. So you'd never be in a problem state of `setq` failing halfway, for example.

Personally, I think this model works much better with the current code in Emacs. It would be good if we could spin up domains like in Racket for true parallelization (which uses channels for communication) as well, but threads are pretty useful already if hard to code with.

I don't believe there are any locks or semaphores at all, but I'm happy to be educated.

EDIT: caveat ofc, that non-Elisp code can be parallel but the filters and sentinels will only run if they can grab the context.

[klibertp]

Locks: https://www.gnu.org/software/emacs/manual/html_node/elisp/Mu...

Semaphores are not there, my mistake; I was thinking about: https://www.gnu.org/software/emacs/manual/html_node/elisp/Co...

That's basically what every other threading library provides in most languages... and it's also what was shown time and again to be very hard to work with directly. Higher-order abstractions are necessary to make parallelism safe and concurrency convenient.

> and atomicity is guaranteed apart from when you use these calls. So you'd never be in a problem state of `setq` failing halfway, for example.

That's true - it looks like Emacs uses a global lock to ensure the atomicity, similarly to what Python does. Also like in Python, you can release that lock from native code (module or core). You cannot touch any interpreter state from other threads, so you need a bit of plumbing to get the results back, but it's possible. I found this: https://github.com/emacs-lsp/emacs/blob/json-rpc/src/json.c very interesting: it's a fork that moves JSONRPC from Lisp to C and out of the main thread. See for example line 1109 and related.

> but threads are pretty useful already if hard to code with.

That's the point: the capabilities are there (mostly), but abstractions are not. Coding with threads, even in the presence of the global lock, is hard, and ensuring correctness is nontrivial. At the very least we should get channels for communication (share by communicating, don't communicate by sharing) between threads and thread pools for executing tasks (like futures in Java or Python, or Task in Elixir). Threads and locks are way too low-level for normal coding. I suspect that's the reason why they're not used more widely, even though they're there for the third(?) release now.

Aside: Racket is actually a nice example of concurrency and parallelism being treated as completely separate concerns. IIRC threads in Racket are call/cc-based green threads, while places are separate instances of the VM that execute in OS-level thread or separate process. Threads provide concurrency and places provide parallelism. It's actually a good thing, I think. Mixing the two is often a major source of errors. Racket also has futures, which are parallel-if-possible primitives that can benefit from parallelism if they don't touch external state - a sort of a middle ground.

In any case: yes, Elisp threads are a good addition to the language, but they alone are not enough to bring concurrency to the masses, so to speak. As a concurrency primitives, and compared to callbacks, they have few advantages and some serious downsides. Emacs still needs a lot of work on the concurrency front. And don't even mention parallelism, that's another can of worms that we don't really need to open :)

[ywei3410]

Thanks for the link; not sure how I missed that completely while going through the thread documentation.

I completely forgot about native modules completely...

I don't think I disagree with anything you've said; though I would add that the fact that not many things use threads means that it's self-fulfilling (for example I have a package in the works and it was _super_ unclear to me about what the semantics of how threads interacted with the existing processes and when you could guarantee certain things ran before others).

Yes, Racket's model is broadly as you've described (and soon ocaml will have a similar model I think?), and imo it works really well.