[felixgallo]

to be fair, the original article is fundamentally incorrect and Erlang is not truly preemptive either. It uses a reduction-counting based cooperative multitasking system which yields at function calls, but a badly programmed NIF can still ruin your day if you don't put it on a dirty scheduler.

[jeffdavis]

You could say that no scheduler is truly pre-emptive, because it can't interrupt a process in the middle of a machine instruction.

Implementing NIFs should be done with the same care you would use adding a new machine instruction to your processor ;-)

[davidw]

There are N ways that badly coded C code loaded into a Unix process can send things to hell in a handbasket. "All bets are off" as they say.

[felixgallo]

In this case just taking more than a millisecond can cause scheduler collapse. So it's a pretty easy mistake to make.

[hamidreza-s]

Although writing C code for NIFs is not a regular task for Erlang developers, it must be done with extreme care because not only a long-running NIF could degrade the responsiveness of the VM, but also when it crashes the whole VM will crash.

However when there is no other options except writing a NIF, there are ways to protect yourself:

1. Your NIF should return less than a millisecond.

2. If the item 1 is not possible, split it into shorter NIF calls.

3. If item 1 and 2 are not possible so you have a dirty NIF. It is a NIF that cannot be split and cannot execute in a millisecond or less. There is an experimental feature in Erlang virtual machine which is called "dirty scheduler". When it is enabled some other schedulers are ready to execute the dirty NIFs, so they won't interfere with the normal operation of schedulers.

4. If item 1 and 2 are not possible and you don't want to use dirty schedulers, the +sfwi emulator flag is available to force normal schedulers to wake up again from the collapse situation.

These items are some solutions to remain in normal scheduling state even in case of writing the native functions in C (NIF), but what the article says is about just Erlang code which is run by schedulers and are preempted with no trouble as soon as they reach the reduction limit.

[strmpnk]

This. Behavior is much better these days but dirty schedulers and ports are still the first places to consider putting C code. Only when you know you've got a solid implementation should you upgrade it to a NIF.

A short note on how hard this is: Until recently in 18.0, there were many BIFs (built-in functions that are part of the VM) that could possibly cause the same scheduler collapses. If the VM developers don't always get it right, the chances that some C code will, is very small. Tools like QuickCheck can help in testing the inputs and outputs but it's hard to setup complex VM stress states and thus very hard to make guarantees about NIFs.

While there is definitely overhead, I'd say regular external "ports" (OS-level subprocessing) are quite underrated from what I see in more recent Erlang code. There's a lot that can be done this way if port communication is carefully designed.

[sasa555]

I like to say that in NIFs (and port drivers) all the bets are off anyway, most notably process isolation and fault-tolerance. The guarantees such as "preemption" and fault-tolerance happen in the layer above (i.e. pure Erlang), and can only be satisfied if the underlying layer behaves properly.