It's a funny interaction. While I was mad initially, GPT-4 creating the issue actually solved problem for the user, so yeah I don't know if this should be counted as a positive or negative example of AI.
Here it's not really blowing past a guardrail, but rather it's a sharp corner the end user didn't expect.
End user set it up with tools that told ChatGPT -- If you need to open an issue, here's how: zzzzzzzzzz. Then he asked ChatGPT a question and was surprised that it did zzzzzzzzzzz and opened an issue without asking.
Said tools may want to clarify their instructions to ChatGPT-- that users will usually want to be consulted before taking these kinds of actions.
“Human in the loop” is meant to be “a human is always in positive control of the system’s actions.”
It does not mean “system will sometimes do things unexpectedly and against user’s intention but upon generous interpretation we might say the human offered their input at some point during the system’s operation.”
Exactly, this is not human in the loop. The plugin was created without guard rails. A human in the loop guard rail would be "here is an issue template, please confirm to post this". It's really a simple change and this is the sort of thing that regulation should address, it shouldn't try to ban the technology outright, but rather require safe implementation.
At the same time, the degree of guard-rail necessary in the plugin is unclear. Is opening a GitHub issue something that should require user confirmation before the fact? Probably, but you could convince me the other way-- especially if GPT4 gets a little better.
We decide how much safety scaffolding is necessary depending upon the potential scale of consequences, the quality of surrounding systems, and the evolving set of user expectations.
I'm not sure regulators should be enforcing guard-rail on these types of items-- or at least not yet.
Assign blame wherever you want, the fact of the matter is this is not what most people mean when they say “human in the loop.” The “AI will always have HITL” argument was always weak, but now plainly disproven.
The logged behavior would surprise many totally sensible people, as you’re seeing in this comment thread.
What exactly was the user error? Are we to believe that if you authenticate a plug-in into your session you are okaying it to do any of its supported operations, even at wildly unexpected times, and this is considered “in the loop?”