[curiousthought]

This is actually a great use case for something like Windows Recall. Ingestion of data after the fact requires the data to be discoverable.

If there was a way to add a meta-prompt to Windows Recall like "Create a log entry every time I watch something with its title and URL" it could serve as a history whether things were watched on YouTube, Vimeo, or any other site, without requiring plugging into each service individually. Repeat ad nauseum for each thing to be logged, or perhaps someone can come up with a more clever query than I that catches everything sufficiently.

The level of granularity on many services might be surprisingly large, preventing introspection of the data at a useful level.

[groby_b]

This is a horrible use case for Windows Recall. Even if we ignore all the privacy implications of having a third party screenshot you every 30 seconds and making the files world readable, it's a bad idea.

Recall has lost a ton of useful metadata you already have - both URL visits and streaming are clearly discernible actions, both at the network stack level, and from your browser history. Throwing that away to trust an LLM to re-infer the same data is both reducing data fidelity and significantly increasing processing cost.

If you want to see this done reasonably well, I'd suggest looking at e.g https://beepb00p.xyz/promnesia.html (which not surprisingly bears a strong similarity to what the article discusses)

LLMs don't add much value here, outside of tightly locked down systems where screenshots are the only way of exporting.

[curiousthought]

Sorry when I said something like Windows Recall, I didn't mean Windows Recall but software with similar capabilities. I think in my mind I was imagining some sort of ongoing screen capture along with a meta prompt or prompts, and some sort of output.

The value the LLM adds is interpreting/processing data without having to tailor input streams. Imagine if formats change, fields get renamed, and so on. The maintenance would be a headache if this was done on a per-service level. I think the reduction in fidelity seems like a reasonable tradeoff, but that's for the user to decide of course along with local/cloud processing and proprietary/open source software.

Even things like invoices from the same service change format over time.

[netsharc]

I've been using https://www.manictime.com for maybe close to 20 years now, although not the pro version that offers screenshot recording (curiously the website doesn't mention the existence of a free "standard" license). It records window titles and presence/away times.

A prompt every few minutes that would ask "What are you doing now?" would be interesting to me, as a professional procrastinator. Maybe an even better one would be one that says something like "In the last 10 minutes, you spent 90% of it on Hacker News".

[groby_b]

The (non-privacy related) issue is the same - if you resort to screen shots, you've thrown away tons of valuable metadata.

And, as long as there's an API, I am fairly certain that maintaining a compat layer is a lot less work than retuning the LLM when the images change. (And you'll want to adjust your tuning, at least with current SOTA, or your error rate will reach Unpleasantville fairly quickly)

Yes, it seems easier on the face of it - but the reality of building an LLM pipeline will quickly point out a lot of edge cases.

[joshcanhelp]

I've been talking about this with a friend/colleague of mine. Some of the greatest sources of data are "recorders" that are running in the background. Oura or any other activity tracker, Google Maps with location history turned on, browser history. Data privacy concerns aside, those sources can really help tie things together.

Personally, I stopped using my Oura ring and don't have location history collected anywhere (that I control or know of). One of my big ideas for this is a native application for your phone and laptop that collects everything it can in the background and surfaces it somewhere you control (synced via a service you choose, lands on a machine you own). Maybe overkill for many people but being able to access that without giving it to another service would be something I'd use.

[IncreasePosts]

Browser history does that pretty well currently.