[sgarland]

I wonder if distros like Tails [0] are going to start shipping lightweight LLMs specifically to reword messages. Though I’m also not sure how low of a spec you can go and still run an LLM decent enough to not be excruciatingly slow.

[0]: https://tails.net/

[seanhunter]

You don’t need a special distro to run a local llm and something like ollama running llama3 7b would be just fine to reword a message on a normal laptop. Inference (ie actually using a model) is much much less compute intensive than training or finetuning.

I strongly suggest people try ollama - it takes a few minutes to set up, download a local model and you’re up and running. https://ollama.com/

[sgarland]

I know – I've been playing with Ollama on my MBP and it's great.

My specific mention of Tails was because it's designed as an ephemeral OS for the paranoid / extremely security conscious, so it would make sense that they would consider something that allows further obfuscation of a user's identity.