[curiousthought]

Sorry when I said something like Windows Recall, I didn't mean Windows Recall but software with similar capabilities. I think in my mind I was imagining some sort of ongoing screen capture along with a meta prompt or prompts, and some sort of output.

The value the LLM adds is interpreting/processing data without having to tailor input streams. Imagine if formats change, fields get renamed, and so on. The maintenance would be a headache if this was done on a per-service level. I think the reduction in fidelity seems like a reasonable tradeoff, but that's for the user to decide of course along with local/cloud processing and proprietary/open source software.

Even things like invoices from the same service change format over time.

[netsharc]

I've been using https://www.manictime.com for maybe close to 20 years now, although not the pro version that offers screenshot recording (curiously the website doesn't mention the existence of a free "standard" license). It records window titles and presence/away times.

A prompt every few minutes that would ask "What are you doing now?" would be interesting to me, as a professional procrastinator. Maybe an even better one would be one that says something like "In the last 10 minutes, you spent 90% of it on Hacker News".

[groby_b]

The (non-privacy related) issue is the same - if you resort to screen shots, you've thrown away tons of valuable metadata.

And, as long as there's an API, I am fairly certain that maintaining a compat layer is a lot less work than retuning the LLM when the images change. (And you'll want to adjust your tuning, at least with current SOTA, or your error rate will reach Unpleasantville fairly quickly)

Yes, it seems easier on the face of it - but the reality of building an LLM pipeline will quickly point out a lot of edge cases.