[darby_eight]

IDK about "lazy", but it's certainly an extremely expensive solution.

[dugite-code]

Isn't the expensive part of LLM's the training? My understanding is once they are trained they can often be optimized to run quite cheaply. Not as cheaply as a well designed program but cheaply enough it shouldn't be too prohibitive to run.

[denton-scratch]

I'd love to be shown I'm wrong; but I thought most 'runtime' LLMs required a shit-ton of memory. Just downloading one seems to require more storage than I have on this laptop.

[refulgentis]

A 3B model runs on Android phones from 2 years ago at 6 tkns/s.

[darby_eight]

I'm not sure what you're comparing this to or how you're making this comparison—can you enlighten us?

(Somehow I doubt whatever caldav software the above poster references takes more than a second to process multiple emails.)

[throwaway11460]

I don't think it's a comparison - they're just saying that it's fast enough even on old mobile hardware, so it can be used on new hardware even faster.

I don't have a problem with a background task taking a minute or something...

[refulgentis]

Cost isnt a relevant factor for this