[imtringued]

How is that cheaper? You now need to have a database of millions of possibly gigabyte sized rows. Also, transformers have quadratic complexity, so short queries cost practically nothing.

The only optimization that makes sense is per user prefix caching, because you are often sending the same system prompt over and over again or are continuing a conversation.

[joegibbs]

Why would the rows be gigabyte size? You’d just save the text.