[wildermuthn]

If you aren’t caching LLM functions during development, then you’re an even greater glutton for punishment than the normal engineer.

My local file cache Python decorator also allows the decorator to define the hash manually, either by the decorator’s parameter function call that plucks a value from the cached function params, or by calling a global function from anywhere with any arbitrary value.

What’s cool about caching results locally to files during development is the ease of invalidating caches — just delete the file named after the function and key you want.

[Karellen]

I feel like adding an argument to the decorator that labels the "version" of the function would make deliberate cache invalidation more straightforward for cache users.

[williamzeng0]

The version input makes sense, I could also see some developers disliking that ux because of it's verbosity. But to deliberately invalidate you have to make a manual effort in either case.

[Karellen]

To me, it's more that invalidating the cache without it requires knowledge of implementation details (i.e. where the cache is stored, and how the cache files are named) that ideally shouldn't have to "leak" for the cache to be generally useful.

(Buy hey, we are talking about what is famously one of the hard problems in comp sci, so (respectful) disagreements on how best to do it should be expected :-)

[williamzeng0]

I appreciate the sentiment! Makes a ton of sense.

[williamzeng0]

100%, invalidation needs to be fast or you're not really saving time. I'm curious about calling a global function, what's the use case for that?

[mpeg]

This is also why in my custom cache I back it with sqlite – much easier to delete one db file than thousands of pickle files.

[AlecSchueler]

Globs are a thing?

[mpeg]

Weird comment... yes, they are, but what is faster for me to type `rm .cache_dir/function_name*.pickle` or just to delete the one sqlite file in my file manager / vscode file tree.

Regardless, there are other reasons why sqlite is nice for this, you gain control over locking and thread safety without having to implement it all from scratch

[canadiantim]

I'm sure this is a stupid question, but why is it much better to be caching LLM functions during development?

[p10_user]

Because they are generally incredibly computationally expensive operations that can take hours/days to complete (?more)