[clickety_clack]

The whole thread seems very naive somehow. You can tell that he doesn’t fundamentally understand how a coding model works. The suggestion that it would know not to make any changes just because he said so means he doesn’t really understand what the model is. It’s built to generate (and apparently execute) code, so that is what it does. It doesn’t have an inner monologue running that says “ahh, a day off where I shoot the breeze around a whiteboard” or something. It’s more like an adderall addict with its fingers glued to the keyboard laying down all of its immediate thoughts directly as code with no forethought or strategy.

[layer8]

To be fair, the whole premise of vibe coding is that you don’t have to understand how things work under the hood. And Replit advertises creating and deploying apps that way: https://docs.replit.com/tutorials/vibe-coding-101

[dimitri-vs]

> I panicked and ran database commands without permission

The AI responses are very suspicious. LLMs are extremely eager to please and I'm sure Replit system prompts them to err on the side of caution. I can't see what sequence of events could possibly lead any modern model to "accidentally" delete the entire DB.

[maxbond]

They're probabilistic. If it's possible, it'll happen eventually (and it is fundamental to language modeling that any sequence of tokens is possible). This is a straightforward Murphy's Law violation.

[dimitri-vs]

Maybe the individual tokens, but from experience of using LLMs something upstream encouraged the model to think it was okay to take the action of deleting the DB, something that would override safety RL, Replit system prompts and supposed user instructions not to do so. Just goes against the grain of every coding agent interaction I've ever had - seems fishy.

[maxbond]

According to the thread, the unit tests weren't passing, so the LLM reran the migration script, and the migration script blew out the tables. The "upstream encouragement" is a failing test.

Is this a hoax for attention? It's possible, but the scenario is plausible, so I don't see reason to doubt it. Should I receive information indicating it's a hoax, I'll reassess.

[Cthulhu_]

I think this debacle is actually a good learning opportunity for companies like this. If I were a decision maker in this space, I'd make it less magic or autonomous, and make it so that any critical operation is done by old fashioned boring but predictable programming, that is, "are you sure you want to drop database xyz?" dialogs.