[finder83]

There's a component that seems to be missing here which is preemptive task scheduling. I've not seen another non-OS system do it like the BEAM VM (the VM behind Erlang), though there may be something I'm not aware of. It really prevents a whole class of concurrency issues where a hung process can freeze or slow down the entire system.

For example, if you recreated gen_server in a cooperative concurrency environment, one gen_server could use up all of the CPU and have an impact on the performance of the rest of the system. Maybe the other threads (microthreads, not OS threads) would still respond in <500ms, but if every request takes 500ms when they normally take 15ms you could essentially have outage-like conditions, particularly with upstream timeouts.

Instead, because BEAM is preemptive that one (or 10) hung gen_server doesn't hang up everything else on a node. Sure at some point performance will degrade, but that point is much further down the line than in cooperative concurrency models. There was a fantastic talk by Sasa Juric that demonstrates this in Erlang. [1] Otherwise you run a higher risk of even the supervisors being starved for CPU time, particularly if you are launching hundreds of processes that are all locking the CPU.

It's really the combination of the behaviors (OTP), the scheduler, lightweight threads, message passing, and immutability to me that makes the Erlang (Elixir for me) concurrency model so appealing.

Creating a language with the feel of a lisp, the environment of Smalltalk, and the concurrency of Erlang has been my dream for a long time.

[1] https://www.youtube.com/watch?v=JvBT4XBdoUE

[travisgriggs]

> Creating a language with the feel of a lisp, the environment of Smalltalk, and the concurrency of Erlang has been my dream for a long time.

Hey! Now there’s two of us! It would be interesting to compare notes. :)

After 20 years of really dedicated Smalltalk evangelism, I jumped out of that balloon a little over 10 years ago. Then I wandered in many strange lands embracing the polyglots “right tool for the right job” mantra. What I found was more like “here’s a lot of really mediocre tools; rarely is it crystal clear which if many is right for the job.”

A year or so ago, I built out an API server in Elixir (no Phoenix) and I’ve really loved it. Great community. I love that it’s built on basic fundamental principals, and not a bunch of edge cases. I’ve always wondered what some sort of mashup would look like. If I was independently wealthy I would tinker away at such a thing.

[finder83]

"out of that balloon" -- I see what you did there. :)

I've not really thought about it beyond that it would be nice. I really wanted to love Smalltalk, but Pharo at least didn't run well on Arch for me, and I didn't much love the web frameworks/concurrency story. I will say that Seaside is probably what got me interested in Liveview in the first place though. So many "if only"s...the developer experience is amazing in Smalltalk though.

There's a lot that scares me about making languages and I've not studied it much. I've started reading through SICP though as a starting place. Writing a basic scheme interpreter (and future compiler/JIT compiler) as an image-based language (with change tracking) to make it interactive from the get-go seemed to be as far as I made it, maybe basing it on an existing Scheme, but tacking on preemptive scheduling sounds hard. But so is making a language and entire development environment. :)

Shoot me an email though if you'd like to chat (in my profile, also [e-mail]), not 100% sold if this is something I'd want to take on in the future or not, I really wish it already existed and someone better than me had already made it. :-D

[qohen]

>> Creating a language with the feel of a lisp, the environment of Smalltalk, and the concurrency of Erlang has been my dream for a long time.

Have you looked at LFE (Lisp Flavored Erlang), by one of Erlang's co-creators, Robert Virding? (No Smalltalk-like environment, but 2 out of 3 ain't bad, right? :-) )

https://lfe.io/

[kaveh808]

>> Creating a language with the feel of a lisp, the environment of Smalltalk, and the concurrency of Erlang has been my dream for a long time.

>Hey! Now there’s two of us! It would be interesting to compare notes. :)

Please make this happen. :)

[travisgriggs]

Sure. Just need that “independently wealthy” thing to fall in place. Are you donating?

[kaveh808]

I have a cunning plan to become independently wealthy. Should come to fruition real soon now. :)

Perhaps an IDE for LFE would be a place to start? Does such a thing exist?

[trashburger]

> Creating a language with the feel of a lisp, the environment of Smalltalk, and the concurrency of Erlang has been my dream for a long time.

I'm trying to eventually accomplish something like this: https://github.com/sin-ack/zigself

It's an implementation of the Self programming language in Zig, with an actor model inspired by Erlang.

The main thing to realize is that Lisp and Smalltalk are very much symmetrical in terms of structure. There is no real distinction between the two other than syntax and basic computation unit (closures vs. objects). And even closures can be used as objects and vice versa.

That only leaves the concurrency model. I have a basic implementation of actors using objects as the "context". It still has a long way to go to reach the supervisor tree model of Erlang, but interestingly enough, the ideas in the article are reflected here heavily; behaviorism is at the core of Self.

[stevan]

> There's a component that seems to be missing here which is preemptive task scheduling.

For Erlang, yes. For implementing behaviours (the point of my post), I don't think so (I sketch a "single threaded" solution towards the end).

I think one worker behaviour per CPU/core per stage in the processing pipeline is better than throwing thousands of processes at the problem and let the scheduler deal with it. This is what I got from Martin Thompson's talks (I linked to one of them in the "see also" section).

[samsquire]

I wrote a preemptive 1:M:N scheduler in C, Rust and Java.

https://github.com/samsquire/preemptible-thread

It is a 1:M:N scheduler where there is one scheduler thread, M kernel threads and N lightweight threads. I take advantage that loop indexes can be structures and can be modified by other threads. So we can set the thread's looping variable to the limit to end the current loop and pause it and then schedule another thread.

[waynesonfire]

Thanks for pointing this out. It is very important. The preemptive task scheduling is the secret sauces that makes Erlang unique. For example, if you look at how Akka implements the actor system, you'll see it riddled with the Future and/or async/await patterns which is due to the non-preemptive java scheduler. After playing with Erlang, this hack becomes a irritating.

[Fire-Dragon-DoL]

If I'm not wrong, Go added uncooperative task scheduling that behaves similarly?

[ksbrooksjr]

I'm not a Go expert but preemptive scheduling of goroutines is supposed to be one of the primary benefits of Go as a language [1]. Although I'm not sure if there's any runtime that adopts preemptive scheduling as fervently as the BEAM. Even the regular expression engine is preemptive in Erlang, which prevents regular expression denial of service attacks [2].

[1] https://go.dev/src/runtime/preempt.go

[2] https://www.erlang.org/doc/man/re.html#:~:text=run/3%20alway....

[ogogmad]

Regex DoSes can be made impossible without using any concurrency. See Go's and Rust's implementation. Also see https://swtch.com/~rsc/regexp/

[ksbrooksjr]

I wasn't trying to imply that concurrency is the only way to mitigate ReDos attacks, I was just pointing out that the BEAM applies preemption seemingly everywhere, which I found interesting.

[toast0]

The BEAM misbehaves when a scheduler gets stuck in a process (or other activity) for too long. As a result, anything that can take a long time needs to have yield points. This is a human process; when I started using Erlang, garbage collection didn't yield, List1 ++ List2 took forever with large lists and didn't yield, we didn't have line numbers in backtraces, and we had to hotload our code uphill in the snow both ways. It's not unusual for a new OTP release to have added new yield points in BIFs or NIFs that could run long or even in core VM workings.

[ogogmad]

There was someone recently on HN who didn't know about the possibility of non-backtracking implementations of regex.

[Fire-Dragon-DoL]

It wasn't non-cooperative pre-emptive scheduling for a while, which is why I kept pushing Elixir, until Go implemented it. Now Go has the same benefit as Elixir, but it's very, very fast.

[ksbrooksjr]

Yeah with preemptive scheduling, static typing, and better cpu utilization Go is a pretty compelling option for Erlang and Elixir users.

[cztomsik]

It's not immutable, there is no messaging, no hot-swap, no supervisors, it's not FP, and it doesn't have transparent support for clustering.

Also, it's not memory-safe, 30ys old, and battle-tested pretty much everywhere.

[ksbrooksjr]

There are definitely trade offs when switching from Elixir or Erlang to Go. If you're a functional programmer who can't live without immutability, or you plan on running a cluster of machines that can communicate with each other and hot-swap code into the running system, then Elixir and Erlang are good choices.

If you have some extremely CPU intensive code, or you like cross compiling to a lightweight binary, or you need static typing (without giving up preemptive scheduling), Go is a decent choice.

Elixir and Erlang are not slow by any metric, but there are faster languages. Discord famously had to augment their Elixir code with Rust for example to scale to 11 million users [1].

[1] https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...

[Fire-Dragon-DoL]

https://stackoverflow.com/a/73932230/312907 I found this answer.

I can't find the original announcement, but I found the accepted Go proposal: https://github.com/golang/go/issues/24543

I remember that being the straw that made me drop Elixir (which I love, to be clear). Go excels in many, many places where Elixir does, but it's way faster.

I do think Elixir has immutability on their side, which is huge for new developers, but there are way less developers in Elixir than Go, so the end result doesn't change unfortunately.

[finder83]

Thanks for sharing! That's really exciting, going to have to make sure we're on the latest Go version everywhere.

[finder83]

Interesting, at least when I was using it the goroutines themselves were cooperative, and of course the OS threads were preemptive when GO_MAXPROCS > 1. Not finding much with a search, there was a proposal to make them preemptive. Curious if others will chime in that it's now preemptive. Even so I like Elixir better as a language, but the processing speed of Go with a preemptive scheduler would be tempting for some use cases.

[vbezhenar]

Does Erlang prevent a thread from consuming lots of memory and starving other threads? Managing CPU does not seem to be particularly hard. Managing RAM with GC and objects moving between threads is interesting.