[comfypotato]

Since you mentioned the single-threadedness (and it’s relevant to the discussion of IDE/Emacs): what’s the status of any initiative to move Emacs to multiple cores? I’m single, and I hope to one day have a family etc.. If I end up staying single for long enough, helping move Emacs onto multiple cores is something I fantasize about contributing back to the community that has helped my life so much.

Does anyone have any input?

[tmtvl]

Emacs has support for multithreading (threads landed in 26), there was an article about the multithreaded model earlier this year (https://news.ycombinator.com/item?id=31559818), and the Elisp manual has a section on concurrency (section "Threads").

If you want to contribute to the future (pun intended) of multithreaded Emacs, your best bet may be to make a library that adds the nice concurrency facilities that will help library authors move to a multithreaded model. An actor model, green threads, or supervision trees may be avenues worth pursuing.

A big blocker is probably the dynamic binding, but trying to "fix" that would break a lot, if not all, of the existing libraries.

[medstrom]

Please DON'T use the threads. As of 2018 it has many data races (https://nullprogram.com/blog/2018/05/31/), and resultant bugs will be a nightmare to track down.

The Emacs community has always got by through finding other ways to ensure responsiveness. It's true that each individual user shouldn't have to spend that much time on configuration (Doom Emacs and such distros help here), but still: people who find it slow must have misconfigured something, and on profiling, they'll find at most 1 or 2 features of their config that's eating all the CPU cycles. These won't actually be hard to fix, for a programmer, and threads wouldn't have helped matters, and at best only have hid the CPU-eater.

[tmtvl]

Yes, that post was mentioned in the article I linked:

Even Emacs current threads library suffers from data races which is one of the reason I believe it has not seen much adoption.

(where suffers from data races is a link to Chris Wellons' post).

The idea now is, as Chris states:

There really needs to be a safe, high-level API with clean thread isolation. Perhaps this higher-level API will eventually build on top of the low-level threading API.

While you are right that for 99% of standard usage single-threaded code is entirely fine, not utilizing between half to 15/16ths of potential processing power is a bit of a shame, especially considering it means doing something like indexing an entire project (yes, we can use GNU Global) will require the user to just sit and wait.

[medstrom]

That "99% of standard usage" is the kicker, isn't it? The greybeards who opposed multithreading since the beginning tend to say that the remaining 1% of use cases is best done in an external process, ideally not even written in Emacs Lisp, so that Vimmers and the rest of the free software community can benefit. The GNU Global you mention embodies that ideal exactly.

I suppose if you still want that program to be written with Emacs Lisp, you could use async.el (https://github.com/jwiegley/emacs-async/) and there's finally an use-case for the threads: it'll be relatively safe to run those 16 threads only in the external Emacs-process.

[comfypotato]

Thanks for the pleasant surprise of encouraging feedback. I’m going to send this comment chain to my email inbox so that I can find it later. This is great.

It’s such a monumental task to hack on the basics of the text editing. I listened to a tts book; I think it was titled “The Art of the Text Editor” that delved a bit into some of the original philosophy of the text rendering etc. in Emacs. You might enjoy it If you ever need a boring book to read.

[tmtvl]

You probably mean The Craft of Text Editing -or- A Cookbook for an Emacs. I've skimmed it a bit before and it's on my reading list. First I'm still working my way through CRLS's Introduction to Algorithms and Aspnes' Notes on Data Structures and Programming Techniques.

[bloopernova]

Links to the electronic versions of this out of print book: https://www.finseth.com/craft/

[bloopernova]

Warning, low self-esteem and anxiety ahead:

I really wish I had the capacity and capability to pursue this effort. I barely have the energy to be good at shitty DevOps and Terraform, writing basic Python and Node is so time consuming. I can do some Lisp, but only at the basic level of init.el and adapting a few configuration examples.

I do the good thing and donate to the FSF monthly but I wish I could donate directly to Emacs because I use it every day. Even better would be to sponsor the development of a concurrency library and port of major things like magit or org.

[chimprich]

> I wish I could donate directly to Emacs

I've requested this whenever I've had the chance. I would be very happy to donate some cash to get core Emacs improved. Emacs has made a huge positive difference to my professional life for years. Unfortunately this is not made easy.

I think one of the issues last time I looked into it was that FSF couldn't have donations on a platform that was potentially tarnished by having e.g. non-free javascript running on it somewhere... All very correct and in tune with their mission I'm sure, and part of the reason why Emacs etc. has been so successful, but it would be nice if they were a bit more pragmatic occasionally.

[db48x]

Most of what Emacs does just won’t benefit from massive parallelism. CPUs are ridiculously fast at processing text (AVX2 instructions can process vectors of 32 bytes at a time), and most of the stuff that _can_ benefit from parallelism should just be moved to an external process. For example, querying your compiler for a list of methods that apply to the current object, or a list of functions that start with “Foo” are mostly moving to external processes using LSP as the communication protocol. It would be nice for some things to use background threads, like Gnus polling your IMAP server for email, and Emacs does have the threading mechanisms to handle that now. It’s just a matter of time before someone does the work :)

[the-lazy-guy]

I disagree. When I am running a compilation (with output being dumped into a visible buffer) + query magit for large commit. Over tramp. Things noticeably freeze. Technically it all is async. Practically, it is implemented as polling things on main thread with some witing happening in non-async fashion.

> For example, querying your compiler for a list of methods that apply to the current object, or a list of functions that start with “Foo” are mostly moving to external processes using LSP as the communication protocol.

That's why we have lsp-bridge and lsp-mode emacs fork :) Both of which build some infrastructure to avoid doing communication work with lsp-mode work in main emacs thread. So, heavy emacs users are building some async machinery which wraps another already async and relatively lightweight protocol, because core emacs facilities can't keep up with it. Architecturally it is kind of insane.

I think, lsp-mode fork is doing the right thing (from practical POV; it goes against "emacs is just an elisp interpreter" ideology though) and hope it gets into core at some point. A better solution would have being having first class async and background threads support at the elisp level. Which would never happen due to elisp messiness.

https://github.com/emacs-lsp/emacs https://github.com/manateelazycat/lsp-bridge

[comfypotato]

Hey! Exactly the response I was looking for. Thanks for this.

Your response reminded me of an Intel project about regular expressions. I think it’s called hyperscan, but I’m not sure. It’s a regex engine (I think that’s the correct terminology) that makes dedicated use of very specific Intel instructions to reach incredible regex speeds on Intel processors. Would be cool if Emacs could make use of this tech when configured properly. I bet there are some internals that rely on regex.

[ykonstant]

Does emacs use specialized instruction sets like AVX2?

[db48x]

Alas, no. But the power is right there, all we have to is stretch forth our hands and use it.

[lallysingh]

Ha! No. Native compilation just snuck in a little while ago. SIMD will take... longer.