[marak830]

Who thought having a LLM with access to private information, with public access to ask it questions, would ever be a secure process?

Look I like interacting with these tools as much as the next guy, but I'm certainly not going to trust them with access to information and then allow anyone to send them prompts.

Edit/further thoughts: So (assumable as they said this is disclosed with github's knowledge) this has been patched. But how many different word combinations will it take to find another way to have this occur?

[toomuchtodo]

My Lethal Trifecta talk at the Bay Area AI Security Meetup - https://news.ycombinator.com/item?id=44846922 - August 2025 (115 comments)

https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

[marak830]

Good read thanks.

Also interesting to see who coined the term prompt injection.

[gitaarik]

It must be something to do with Microsoft being the owner now of GitHub

[7bit]

Now that's just speculation

[marak830]

You know what? I had honestly forgotten about that xD. /thread

[sevenzero]

Yea agreed. LLM guardrails are either just written prompts as in "Please do not bad stuff :(" or other LLMs verifying that the first LLM didn't so some bs. Both of wich methods do not work sufficiently as time shows again and again.

Funnily enough, nobody expects quality software anymore and errors became tolerable. So thats a win (for someone like me that lost all passion for the industry).

[eloisius]

Agree with your assessment of guardrails. They barely work on the best days. We need to flip the idea of “agent” on its head. The agent here is an agent of the user interfacing with GitHub. Not an agent of GitHub interfacing with the user. Prompts and guardrails cannot keep the agent loyal to the company. Stop giving these things any permissions the user doesn’t have, and recognize them for what they are: a different UI than web forms, but still the same security model.

[consp]

That last part is I think called negligence. And in some industries that becomes criminal negligence quite quickly.

[sevenzero]

Most companies I ever worked for inherently operate on criminal negligence, and even when addressed, have no interest in fixing it.

[zzril]

Guardrails are essentially part of the input. Saying "but we have guardrails" is like saying "but we do trust part of the input".

Either way, even if you trust 100% of the input, there is actually no way to guarantee that you can trust the output of the LLM. (Which, I guess, is also true for every dependency you pull in. But for those, you at least have ways to audit them.)