The semantics of Erlang and Elixir are very similar, as they're both dynamically typed functional languages, so the performance difference should be negligible.
They are not only similar they are in fact the same. Elixir compiles down to Erlang and the Elixir libraries are built using Erlang and OTP "underneath". Also the the BEAM, the Erlang VM, is designed to run Erlang so it is difficult to make an efficient implementation of a language with different semantics on top of it. It is truly Erlang all the way down.