[bonzini]

How does it do that? When the 64-bit pointer is actually a pointer, does it mangle it into something that is certainly not UTF-8 (e.g. changing its top byte to 0xC0, and assuming that bits 56-63 are the same as bits 48-55, or something like that)?

[tialaramex]

UTF-8 is a self-synchronizing encoding, so each UTF-8 byte tells you whether there were any previous bytes you'd need in order to understand it. Since we're storing a whole string our first byte logically cannot be such a "continuation byte", if it seems to be, we can say these eight bytes aren't UTF-8 encoded text but instead a pointer. The UTF-8 "continuation" marker is that the top bit is set and the next bit is not. So, we need to smash two bits of our pointer.

Now, by pointing only at memory aligned to 4 bytes, the way a 32-bit integer would be on a typical machine, the bottom two bits of the pointer are always zero. So, we ask for such 4-byte aligned pointers when allocating memory to point at, most allocators actually never give out smaller alignments anyway because they cause problems in C but we've told the allocator our requirement so even if it could it won't give out unaligned pointers.

Finally we need to re-arrange our pointer so that those two zero bits are in the right place to put a UTF-8 continuation marker. In Rust this is legal safe code, the map_addr method on a pointer is allowed to twiddle with a pointer's address bits (on typical platforms, all of them) and so long as your transform is reversible (which this one is) it will work correctly when you twiddle the pointer back and dereference that pointer in unsafe Rust.

[bonzini]

Ah, so it is basically a single rotr(x|2, 2), nice