[Rhapso]

FTR LLM embeddeds are a lot less efficient than ASCII for communication and are likely to stay that way.

[riku_iki]

absolutely not. Transformer layers already communicate using embeddings, and ASCII would be absolutely less efficient there.

[Rhapso]

And how many bits are in an embedded vector?

[riku_iki]

12k for gpt3.

[riku_iki]

It is not bits, but weights

[Rhapso]

So somehow ascii is less information dense than 12k 32-bit floats per token?