Hypothetical: Could a model self worm an agent system?
Jetbrains itself doesn't really write any code, nor does it have any range on interpreting what you're asking it. You can't really say "Jetbrains, write an HTTP scraper". With an LLM you can say "write HTTP scraper" and the output of this command might be a HTTP scraper, it also might be a crypto wallet stealing worm.
This is why your simple view of liability falls apart. On most machines you can expect a particular set of actions to have a particular set of outputs. Most machines you can take apart and map what will occur. With an LLM you cannot know the output of a prompt until you run the prompt. In theory if you run the same prompt twice you'll get the same output, but even that is not a given. It behaves somewhat more like a human where you can give them a task to do, but if they do something illegal instead said human would take on the liability.
Sure, but in this case we know the user told their llm to go find open source projects to do this and then to write the blog posts. If it did all that unprompted we could talk about model liability I think, but this isn't a case where it was unexpected as far as anyone knows right?
I mean we already have cases where LLMs are getting root via creative and unprompted means. Also the times AI feels like it messed up and preemptively deletes the production database (and yes this was foolish on the human users)
So ya, the particular article case is prompted, but the underlying issue cannot be ignored that LLMs can have behaviors outside of prompt expectations and agentic loops can further exacerbate this.
> you can give them a task to do, but if they do something illegal instead said human would take on the liability
If an employer says "don't break the law" but nonetheless incentivises their employees to break the law, it is the employer who is vicariously liable. A famous example being Domino Pizza's "30 minutes or its free" policy which incentivised their employees to ignore all driving laws in order to deliver within 30 minutes, their wages depended on it. This caused a number of crashes, injuries and deaths. One recent example, even since Dominos removed their policy, is Coryell v. Morris where they found Dominos still exercise control over their franchisees sufficiently to qualify for vicarious liability for the franchisee's employees' actions: https://law.justia.com/cases/pennsylvania/superior-court/202...
There will be a line in the sand drawn in the future. I hope it's drawn so that people offering internet-based services, where they retain ultimate control of what a tool says/does, will be liable for what it says/does.
Jetbrains itself doesn't really write any code, nor does it have any range on interpreting what you're asking it. You can't really say "Jetbrains, write an HTTP scraper". With an LLM you can say "write HTTP scraper" and the output of this command might be a HTTP scraper, it also might be a crypto wallet stealing worm.
This is why your simple view of liability falls apart. On most machines you can expect a particular set of actions to have a particular set of outputs. Most machines you can take apart and map what will occur. With an LLM you cannot know the output of a prompt until you run the prompt. In theory if you run the same prompt twice you'll get the same output, but even that is not a given. It behaves somewhat more like a human where you can give them a task to do, but if they do something illegal instead said human would take on the liability.