This turns out to be much simpler than expected, on paper at least.
First, this is a tracing JIT and so it basically inlines everything. There is no argument passing convention at all within a block of JIT code because subroutines bodies are completely inline. So - passing tagged values as arguments is actually a fairly infrequent operation.
Then in cases where tagged values really do need to be passed around they will always be passed by pointer reference. The JIT will write them to memory - the Lua stack or the heap - and pass a pointer reference in a register.
The really happy circumstance is that the whole C runtime system is already written to reference tagged values using pointers, and there is already a port that supports 64-bit Lua values using 32-bit machine words.
I see. It seems like that would still make calling supporting routines annoyingly expensive, but perhaps that (and the extra tag memory) doesn't matter as much as I think it does.
There are really two kinds of supporting routines here.
Runtime system routines operate on tagged values and do have to swallow the stack convention. Thankfully the code is already written in this style and the JIT seldom invokes any of these routines in optimized programs (it generates inline versions.)
Second is C routines called via FFI. These use native C data types and the standard platform calling convention. They never see the tagged values (get an untagged uint64_t in a register instead of a TValue struct.) These are easy, cheap, and frequently used.
First, this is a tracing JIT and so it basically inlines everything. There is no argument passing convention at all within a block of JIT code because subroutines bodies are completely inline. So - passing tagged values as arguments is actually a fairly infrequent operation.
Then in cases where tagged values really do need to be passed around they will always be passed by pointer reference. The JIT will write them to memory - the Lua stack or the heap - and pass a pointer reference in a register.
The really happy circumstance is that the whole C runtime system is already written to reference tagged values using pointers, and there is already a port that supports 64-bit Lua values using 32-bit machine words.