[rcarmo]

The generalist view is that nobody is using Erlang for modern telco workloads anymore.

(yes, there are exceptions. Bear with me)

I learned Erlang when we were programming Ericsson voice switches some twenty years ago, and even though Nokia also adopted OTP and did some pretty amazing things with it (including a Hadoop analogue that used Erlang to coordinate Python workers across nodes), I have only come across Erlang when looking at legacy workloads that needed to be moved to the cloud.

NFV and CNF efforts (i.e., virtualized and containerized network functions) I've come across over the past few years seem to be mostly Go or C++. Jaeger is a common tool. I see a lot of Postgres. I also see really weird CNI approaches (check out multus, because telcos still can't get over having dedicated network interfaces for things even though it's all just a virtualization sandwich and they really should just learn to use network policies).

The technicalities run deeper than web apps (because some thing are very, very finely tuned), but the key point is that the telco "bleeding edge" landscape is now indistinguishable from "web scale" K8s discussions, except that telco workloads demand fixed resource allocations nd we still rely _a lot_ on CPU pinning of specific functions due to latency/jitter sensitive workloads.

It's almost like you took low-latency, pseudo-real-time stuff from embedded systems and shoved it into K8s. No, wait, it's _exactly_ like that.

Source: I work in Azure for Operators and have been in the telco industry since the mid-90s.

[ztion]

Ericsson is still using erlang for modern telecom workloads, they use it in packet core (SGSN-MME, AMF). With single nodes handling many millions of subscribers.

[rcarmo]

Of course, they have a huge investment in it. But there are loads of new entrants to the space, and the skill set is very rare.

[anonymousDan]

Are you saying BEAM introduces too much latency and/or jitter for telco workloads? This surprises me. I would have thought that if anything BEAM should be pretty efficient for network I/O. Also what would be the causes of unacceptable jitter? The runtime itself is designed to avoid things like long gc pauses - is it that you end up having to migrate elixir processes between BEAM scheduler cores?

[throwaway81523]

I don't think latency and jitter are big issues. If you're imagining VOIP packets flowing through the Erlang app, it's more likely that they are routed by hardware at a lower level. The Erlang app controls the lower level switches. I don't know about Ericsson switches in particular, but I've worked on telecom stuff (not programmed in Erlang, unfortunately) and that's how it worked. The low level routing was done by FPGA's controlled by a C program. The C program could have been written in Erlang instead, and that would have made life a lot nicer for the programmers.

[rcarmo]

You'd be surprised exactly how much functionality is written in vanilla C and pushed down the stack to raw packet handling. There is hardware acceleration in NICs, but voice is actually a tiny, almost insignificant amount of the workloads involved in some parts of the world.

Think hyperscaler-style SDN, but largely on-premises.

[rcarmo]

I said no such thing. Merely that there is a higher reliance on hardware features even if the workloads are virtualized - more so than on your vanilla K8s cluster in any other industry (except perhaps trading).

[Scramblejams]

I used to experiment a bunch with GPU passthrough to VMs and also found that CPU pinning was required to avoid stutters. I always found it unfortunate as pinning works directly against what one tries to achieve with virtualization (i.e. sharing of resources).

I had never considered telecom suffering from the same issue. Do you know if there's been any work to improve scheduling to mitigate this requirement?

[rcarmo]

There are other issues at play, such as the need for SR-IOV support and sometimes direct mapping of PCI lanes to some workloads. There is only so much virtualization can do when you're letting millions of sessions through.

[geodel]

Excellent points. I work in telco though mainly on business web services side but what you said is about same I heard in here on network side.

[the_only_law]

Are there any good reading on modern work being done in telecom networks. I enjoyed this comment and have had a passing interest in the industry, but have been unable to understand modern networks so far.

[rcarmo]

You can try digging into modern 5G stacks. Search for VNF/CNF, be wary of the fact that it is a much deeper industry acronym-wise (heck, we have 4 and 5 letter acronyms that are not jokes), and hang on to your hat...

[Fantarina]

Thank you for your input, I was thinking that with all the 5g decoupling,network functions and messages passing Erlang would go stronger and need more workforce