[purplehat_]

I often see things like this and get a little bit of FOMO because I'd love to see what I can get out of this but I'm just not willing to upload all these private documents of mine to other people's computers where they're likely to be stored for training or advertising purposes.

How are you guys dealing with this risk? I'm sure on this site nobody is naive to the potential harms of tech, but if you're able to articulate how you've figured out that the risk is worth the benefits to you I'd love to hear it. I don't think I'm being to cynical to wait for either local LLMs to get good or for me to be able to afford expensive GPUs for current local LLMs, but maybe I should be time-discounting a bit harder?

I'm happy to elaborate on why I find it dangerous, too, if this is too vague. Just really would like to have a more nuanced opinion here.

[jwr]

> I'm just not willing to upload all these private documents of mine to other people's computers where they're likely to be stored for training or advertising purposes.

And rightfully so. I've been looking at local LLMs because of that and they are slowly getting there. They will not be as "smart" as the big models, but even a 30B model (which you can easily run on a modern Macbook!) can do some summarization.

I just hope software for this will start getting better, because at the moment there is a plethora of apps, none of which are easy to use or even work with a larger number of documents.

[ben_w]

The docs I upload are ones I'd be OK getting leaked. That also includes code. Even more broadly, it also includes whatever pics I put onto social media, including chat groups like Telegram.

This does mean that, useful as e.g. Claude Code is, for any business with NDA-type obligations, I don't think I could recommend it over a locally hosted model, even though the machine needed to run a decent local model might cost €10k (with current price increases due to demand exceeding supply), that the machine is still slower than what hosts the hosted models, that the rapid rate of improvement means a 3-month delay between SOTA in open-weights and private-weights is enough to matter*.

But until then? If I'm vibe coding a video game I'd give away for free anyway, or copy-editing a blog post that's public anyway, or using it to help with some short stories that I'd never be able to charge money for, or uploading pictures of the plants in my garden right by the public road… that's fine.

* When the music (money for training) stops, it could be just about any provider whose model is best, whatever that is is likely to still get distilled down fairly cheaply and/or some 3-month-old open-weights model is likely to get fine-tuned for each task fairly cheaply; independently of this, without the hyper-scalers the supply chains may shift back from DCs to PCs and make local models much more affordable.

[JonChesterfield]

> The docs I upload are ones I'd be OK getting leaked. That also includes code.

That's fortunate as uploading them to a LLM was you leaking them.

[ben_w]

"Leaking" is an unauthorised third party getting data; for any cloud data processor, data that is sent to that provider by me (OpenAI, everything stored on Google Docs, all of it), is just a counterparty, not a third party.

And it has to be unauthorised, e.g. the New York Times getting to see my ChatGPT history isn't itself a leak because that's court-ordered and hence authorised, all the >1200 "trusted partners" in GDPR popups if you give consent that's authorised, etc.

[autoexec]

> I'm sure on this site nobody is naive to the potential harms of tech

I don't share your confidence. A lot of people seem to either be doing their best to ignore the risks or pretending that a nightmare scenario could never happen to them for some reason. They place huge amounts of trust in companies that have repeatedly demonstrated that they are untrustworthy. They ignore the risks or realities of data collection by the state as well.

> I don't think I'm being to cynical to wait for either local LLMs to get good or for me to be able to afford expensive GPUs for current local LLMs, but maybe I should be time-discounting a bit harder?

I'm with you. As fun it would be to play around with AI it isn't worth the risks until the AI is not only running locally but also safely contained so that it can only access the data I provide it and can't phone home with insights into what it's learned about me. I'm perfectly fine with "missing out" if it makes it harder for me to be taken advantage of.

As a side benefit, if/when AI becomes safe to use with my personal information, it'll probably suck a little less, and others will have already demonstrated a number of tasks it's successful/disastrous at so I can put it work more easily and effectively without being burned by it.

[GeneralMaximus]

I've been analyzing my Obsidian vault using local LLMs that I run via Apple's mlx_lm. I'm on an M4 MacBook Pro with 48GB RAM.

The results are ... okay. The biggest problem is that I can't run some of the largest models on my hardware. The ones I'm running (mostly Qwen 3 at different numbers of parameters and quantization levels) often produce hallucinations. Overall, I can't say this is a practical or useful setup, but I'm just playing around so I don't mind.

That said, I doubt SOTA models would be that much better at this task. IMO LLM generated summaries and insights are never very good or useful. They're fine for assessing whether a particular text is worth reading, but they often extract the wrong information, or miss some critical information, or over-focus on one specific part of the text.

[matusp]

I don't really buy this post. LLMs are still pretty weak at long contexts and asking them to find some patterns in data usually leads to very superficial results.

[embedding-shape]

No one said you cannot run LLMs with the same task more than once. For my local tooling, I usually use the process of "Do X with previously accumulated results, add new results if they come up, otherwise reply with just Y" and then you put that into a loop until LLM signals it's done. Software-wise, you could add so it continues beyond that too, for extra assurance.

In general for chat platforms you're right though, uploading/copy-pasting long documents and asking the LLM to find not one, but multiple needles in a haystack tend to give you really poor results. You need a workflow/process for getting accuracy for those sort of tasks.

[dns_snek]

> and then you put that into a loop until LLM signals it's done

And after that? What's next?

[embedding-shape]

Then you have non-superficial results based even on long contexts. Wasn't it clear I was responding directly to parent's concerns? Could I have made it clearer?

[dns_snek]

What I was getting at is that running LLMs in a loop doesn't really address those concerns at all. You simply get more information, it doesn't do anything for the correctness of that information.

Any patterns it identifies could be imaginary, and you don't have any indication of confidence that all relevant trends have been identified. The most valuable patterns are likely going to be the most hidden and easily missed - those are the ones people are paid to find, not just the low hanging fruit that can be identified with basic statistics.

In the same way that asking an LLM to review some code might produce a hundred superficial comments and miss the things that actually matter.

[jennyholzer2]

no one said you can't turn on the radio and start listening to static

[embedding-shape]

Sure. Is there a point you're trying to make by saying this? I'm afraid your comment is so succinct it isn't obvious what you are trying to say.

[ashirviskas]

It really depends on how deep you want to go. And this will likely not be useful in any way, other than a new hobby. Me and my friends who do this kind of thing, we do it for fun.

If it was not fun for me, I would not have bought 3 GPUs just to run better local LLMs. Actual time, effort and money spent on my local setup compared to the value I get does not justify it at all. For 99% of the things I do I could have just used an API and paid like $17 in total. Though it would not have been as fun. For the other 1% I could have just rented some machine in cloud and ran LLMs there.

If you don't have your private crypto keys in your notes worth millions, but still worry about your privacy, I'd recommend just renting a machine/GPU in a smaller cloud provider (not the big 3 or 5) and do these kind of things there.

[yencabulator]

If you have an extra 20 GB of RAM and a recent-enough CPU (no GPU needed), you can run qwen3:30b-a3b locally well enough to analyze documents and have it report back quickly enough to be completely realistic for analytical use. I find the output of Qwen3's 30B model for that sort of task is plenty good enough.