[bicx]

For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood.

For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's running on the same OS process as your Elixir app, you bypass the failure recovery that makes an Elixir/BEAM application so powerful.

Normally, an Elixir app has a supervision tree that can gracefully handle failures of its own BEAM processes (an internal concurrency unit -- kind like a synthetic OS process) and keep the rest of the app's processes running. That's one of the big selling points of languages like Elixir, Erlang, and Gleam that build upon the BEAM architecture.

Because it uses NIFs (natively-implemented functions), an unhandled exception in Pythonx would take down your whole OS process along with all other BEAM processes, making your supervision tree a bit worthless in that regard.

There are cases when NIFs are super helpful (for instance, Rustler is a popular NIF wrapper for Rust in Elixir), but you have to architect around the fact that it could take down the whole app. Using Ports (Erlang and Elixir's long-standing external execution handler) to run other native code like Python or Rust is less risky in this respect because the non-Elixir code it's still running in a separate OS process.

[thibaut_barrere]

One possibility for production use (in case there is a big value) is to split the nodes into one "front" node which requires strong uptime, and a "worker" node which is designed to support rare crashes gracefully, in a way that does not impact the front.

This is what we use at https://transport.data.gouv.fr/ (the French National Access Point for transportation data - more background at https://elixir-lang.org/blog/2021/11/10/embracing-open-data-...).

Note that we're not using Pythonx, but running some memory hungry processes which can sometime take the worker node down.

[h0l0cube]

> an unhandled exception in Pythonx would take down your whole OS

Is there a class of exceptions that wouldn't be caught by PythonX's wrapper? FTA (with emphasis added):

> Pythonx ties Python and Erlang garbage collection, so that the objects can be safely kept between evaluations. Also, it conveniently handles conversion between Elixir and Python data structures, bubbles Python exceptions and captures standard output.

And...

> Rustler is a popular NIF wrapper for Rust in Elixir

From Rustler's Git README:

> The code you write in a Rust NIF should never be able to crash the BEAM.

I haven't used Rustler, Zigler or PythonX (yet), so I'm genuinely asking if I'm mistaken in my understanding of their safety.

[chefandy]

I hadn’t heard of gleam. Looks cool! I like working with elixir in a lot of ways but never was a Ruby guy, and I think I’d prefer the C-style syntax.

[ikety]

My current favorite language, just no time to finish my gleam projects.

[tmountain]

I’ve been eyeing gleam as my next language to learn. Lots to like about it for sure, and I have always like the idea of OTO but never had an opportunity to tinker with it.

[chefandy]

Ah, projects. Ain’t it always the way.

[giancarlostoro]

I'm more of a Python and C# kind of guy, so Elixir never really hit the itch for me, but Gleam definitely does. One of these days I'll take a crack to see how I can use Gleam with Phoenix.

[__jonas]

I’d recommend to first see if you can use a full-Gleam solution (like wisp/lustre) if it’s a greenfield project – interop is of course possible but can sometimes be a bit unpleasant due to the difference in data structures (Elixir structs va Gleam records) and inability to use Elixir macros directly from Gleam, which are heavily used by projects like Phoenix and Ecto.

[chefandy]

I’ve been mostly in Python, C# and C++ for the past decade or so but got into Elixir as my first functional language. Never got comfy with the syntax but dig how everything flows. Looking forward to digging into Gleam.

[neonsunset]

If you liked Elixir but found it too "exotic" you may find F# enjoyable instead - it's a bit like Elixir but with a very powerful, gradually typed and fully inferred type system and has access to the rest of .NET. Excellent for scripting, data analysis and modeling of complex business domains. It's also very easy to integrate a new F# project into existing C# solution, and it ships with the SDK and is likely supported by all the tools you're already using. F# is also 5 to 10 times more CPU and memory-efficient.

(F# is one of the languages Elixir was influenced by and it is where Elixir got the pipe operator from)

[chefandy]

Been on the list for years — I’ll check it out.

[giancarlostoro]

Do any of them communicate with the BEAM? There used to be a Go based implementation of the BEAM that allowed you to drop-in with Go, I have to wonder if this could be done with Python so it doesn't interfere with what the BEAM is good that and lets Python code remain as-is.

[lawik]

There are several libraries that allow a Python program to communicate with an Erlang program using Erlang Term Format and such.

This approach targets more performance-sensitive cases with stuff like passing data frames around and vectors/matrices that are costly to serialize/deserialize a lot of the time.

And it seems to make for a tighter integration.

[alienthrowaway]

> Because it uses NIFs (natively-implemented functions), an unhandled exception in Pythonx would take down your whole OS process along with all other BEAM processes, making your supervision tree a bit worthless in that regard.

What's the Elixir equivalent if "Pythonic"? An architecture that allows a NIF to take down your entire supervision tree is the opposite of that, as it defeats a the stacks' philosophy.

The best practice for integrating Python into Elixir or Erlang would be to have an assigned genserver, or other supervision-tree element - responsible for hosting the Python NIF(s), and the design should allow for each branch or leaf of that tree to be killed/restarted safely, with no loss of state. BEAM message passing is cheap

[bicx]

That's the thing though: a NIF execution isn't confined to the the BEAM process by its nature. From the Erlang docs:

> As a NIF library is dynamically linked into the emulator process, this is the fastest way of calling C-code from Erlang (alongside port drivers). Calling NIFs requires no context switches. But it is also the least safe, because a crash in a NIF brings the emulator down too. (https://www.erlang.org/doc/system/nif.html)

The emulator in this context is the BEAM VM that is running the whole application (including the supervisors).

Apparently Rustler has a way of running Rust NIFs but capturing Rust panics before they trickle down and crash the whole BEAM VM, but that seems like more of a Rust trick that Pythonx likely doesn't have.

The tl;dr is that NIFs are risky by default, and not really... Elixironic?

[alienthrowaway]

> The emulator in this context is the BEAM VM that is running the whole application (including the supervisors)

You are correct - one could still architect is such that the genserver hosting the NIF(s) run in a separate process/VM/computer in the same cluster since message passing is network-transparent, though inter-host messages have higher latencies.