[adultSwim]

Running an LLM every time someone clicks on a button is expensive and slow in production, but probably still ~10x cheaper to produce than code.

[edwin]

New techniques like semantic caching will help. This is the modern era's version of building a performant social graph.

[daralthus]

What's semantic caching?

[edwin]

With LLMs, the inputs are highly variable so exact match caching is generally less useful. Semantic caching groups similar inputs and returns relevant results accordingly. So {"dish":"spaghetti bolognese"} and {"dish":"spaghetti with meat sauce"} could return the same cached result.

[m3kw9]

Or store as sentence embedding and calculate the vector distance, but creates many edge cases