I agree with this. I basically don’t feel comfortable trying to pretend a fancy matrix multiplication can be responsible for anything.
But what if companies that don’t track responsibility outcompete those who do?
In particular what if some perfect AI decision making ends up nailing decisions that maximize the expected reward for the company. And, well, if that comes at the cost of some unmanaged low-probability catastrophic risk, the company doesn’t care because all the decision makers are AI that don’t mind being shut off.
I don’t know, I can imagine that the equivalent of a PIP for an agent would be noticing flaws in its output and then creating additional evals, modifying the harness, upgrading the model, etc. to improve its performance. That’s not SO different from the PIP, except that the LLM doesn’t really care in the same way?
Like the manager being responsible for someone on a PIP.
Obviously they’re different, but it’s interesting to think about what is unique about humans that makes them able to be accountable or responsible in a way that LLMs cannot be. Is it at core just that they can be fired? Or essentially the threat of suffering?
I think just the possibility of consequences that they can give a damn about. An LLM doesn't have feelings no matter how much human-like text it can simulate. There is literally nothing going on between prompts for any given model. They are incapable of worry or any other emotion. Wipe the context clean and the LLM is completely unaware there was ever a problem.
For a slightly less dramatic version - you can fire a human if they consistently do the wrong thing despite being told how to do it better.
Putting a big ball of matrix arithmetic on a Performance Improvement Plan makes no sense.