Hacker News new | ask | show | jobs
by sevenzero 24 days ago
Yea agreed. LLM guardrails are either just written prompts as in "Please do not bad stuff :(" or other LLMs verifying that the first LLM didn't so some bs. Both of wich methods do not work sufficiently as time shows again and again.

Funnily enough, nobody expects quality software anymore and errors became tolerable. So thats a win (for someone like me that lost all passion for the industry).
3 comments
eloisius 24 days ago
Agree with your assessment of guardrails. They barely work on the best days. We need to flip the idea of “agent” on its head. The agent here is an agent of the user interfacing with GitHub. Not an agent of GitHub interfacing with the user. Prompts and guardrails cannot keep the agent loyal to the company. Stop giving these things any permissions the user doesn’t have, and recognize them for what they are: a different UI than web forms, but still the same security model.
link
consp 24 days ago
That last part is I think called negligence. And in some industries that becomes criminal negligence quite quickly.
link
sevenzero 24 days ago
Most companies I ever worked for inherently operate on criminal negligence, and even when addressed, have no interest in fixing it.
link
zzril 24 days ago
Guardrails are essentially part of the input. Saying "but we have guardrails" is like saying "but we do trust part of the input".

Either way, even if you trust 100% of the input, there is actually no way to guarantee that you can trust the output of the LLM. (Which, I guess, is also true for every dependency you pull in. But for those, you at least have ways to audit them.)

link