[HarHarVeryFunny]

> We are far from solving this problem with LLMs, but this doesn't prevent me from thinking of LLMs as a new level of abstraction that can edit and transform code.

That's more anthropomorphism than an abstraction. An LLM talks like a person because it was trained to predict continuations of human speech. That does not make it a person, or a system with intent, responsibility or any other human attribute. They are what they are: text prediction engines.

Perhaps your input to the LLM is "make all the test cases pass", and so it predicts it better do something to make the test cases pass, and does so by deleting the test cases. I guess in the "abstract" sense it did what you asked.

Or, how about the case in the news from a few days ago where an agentic system deleted all the vendor's customer's data, and last 3 months of backups, despite having been EXPLICITLY "told" not to do any such thing. Should we consider "completely fuck the customer" as an "abstraction" of "never delete any data"?