One thing to consider is that Lua is not a programming language that honors the seamless evolution. Essentially there are three flavors of Lua identified by their versions (5.1, 5.2 and 5.3), almost identical to each other but different enough that a migration is unreasonable. If it were not the case, there is no reason that NeoVim uses Lua 5.1 [1] even though 5.3 is the latest and 5.4 beta is on the way [2]. I think this dependency on 5.1 can become a huge problem later.
NeoVim doesn't care about Lua, they care about LuaJit, which doesn't (and probably wont ever) support lua syntax > 5.1 (or 5.2).
>almost identical to each other but different enough that a migration is unreasonable.
There's nothing unreasonable, and all versions of Lua mentioned are backwards compatible, so there's not even a problem about migrating. Going to a new version is as simple as installing it and running your program.
> Going to a new version is as simple as installing it and running your program.
Not true. Each minor version of Lua tended to inject incompatibilities that can't be trivially resolved. Lua 5.2 had made a catastrophic change to the environment [1], and Lua 5.3 was less severe but there are subtle issues with integer-number conversions you won't ever want in your large codebase [2]. And that is not a theoretical matter, we had to stick to Lua 5.1 exactly due to the environment changes even though 64-bit integers would have made our codebase much cleaner. And Lua 5.1 and 5.2 are no longer supported [3] (though as you have said, LuaJit may be supported) despite of all remaining issues.
Python 2.7 was supported for a decade, providing a reasonable environment when Python 3 was not really matured enough. And while it didn't work, Python 3 tried to provide the migration path.
Perl 6 requires the full p5 compatibility as far as I recall. It was designed to coexist.
Lua only provides some C preprocessor flags for partial backward compatibility.
Without Mike Pall, it won't be progressing as quickly, but it's definitely not dead. It can still fill a niche as a best in class embedded language thanks to its amazing JIT. It's used in Redis, mpv, neovim, and a lot of other very popular software projects.
However, I'm hoping to learn more about the internals of the JIT to see if I can take up some of the effort in progressing it. I have a very deep interest in Luajit for some reason. As long as people like me exist, it's not dead.
Which makes it a very good pick for NeoVim. It basically means that the plugins are basically supported forever without the JIT being a moving target. NeoVim also supports Vimscript as an alternative.
Better to use a more established language than reinventing the wheel