[lmf4lol]

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO.

I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc.

Its on you guys!

My learning: Live on the cutting edge? Be prepared to fall off!

[arcticfox]

There was practically no responsibility taken by the author, all blame on others. It was kind of shocking to read.

Anyone using these tools should absolutely know these risks and either accept or reject them. If they aren't competent or experienced enough to know the risks, that's on them too.

[throwaway041207]

And it doesn't even have to do with these tools in the end, this is a disaster recovery issue at its root. If you are a revenue generating business and using any provider other than AWS or GCP and you don't have an off prem/multi-cloud replica/daily backup of your database and object store, you should be working on that yesterday. Even if you are on one of the major cloud providers and trust regional availability, you should still have that unless it's just cost-prohibitive because of the size of the data.

[pixl97]

Like, shouldn't they teach the 3 2 1 rule of backups in school by now?

[gigatree]

The point of the post was to warn other people building with agents, especially using Cursor or Railway, not a public reflection

[dymk]

It was also to put Cursor and Railway on blast and complain about how they should have safeguarded him from putting a gun to his database and pulling the trigger.

[simonjgreen]

Perhaps they should include a warning about learning systems design and architecture too then? It’s very incomplete.

[shiandow]

For a company that puts DO NOT FUCKING GUESS in their instructions they made a heck of a lot of assumptions

- assume tokens are scoped (despite this apparently not even being an existing feature?)

- assume an LLM didn't have access

- assume an LLM wouldn't do something destructive given the power

- assume backups were stored somewhere else (to anyone reading, if you don't know where they are, you're making the same assumption)

Also you should never give LLMs instructions that rely on metacognition. You can tell them not to guess but they have no internal monologue, they cannot know anything. They also cannot plan to do something destructive so telling then to ask first is pointless. A text completion will only have the information that they are writing something destructive afterwards.

[gwerbin]

The thing that seems to bring up these extremely unlikely destructive token sequences and it totally seems to be letting agents just run for a long time. I wonder if some kind of weird subliminal chaos signal develops in the context when the AI repeatedly consumes its own output.

Personally I don't even let my agent run a single shell command without asking for approval. That's partly because I haven't set up a sandbox yet, but even with a sandbox there is a huge "hazard surface" to be mindful of.

I wonder if AI agent harnesses should have some kind of built-in safety measure where instead of simply compacting context and proceeding, they actually shut down the agent and restart it.

That said I also think even the most advanced agents generate code that I would never want to base a business on, so the whole thing seems ridiculous to me. This article has the same energy as losing money on NFTs.

[mike_hearn]

I don't think it's that. It's really all about context. Humans always have at least a bit of context so it's hard for us to imagine what it's like to have none at all. But the AI genuinely has none. And it's under (training) pressure to get the task done quickly, be a yes man, and so on.

Humans do make mistakes like these. I'm not sure where the fault really lies here. I can imagine a human under time pressure making the same error. It's maybe a goof in the safety design of railway. It shouldn't be possible to delete all your backups with a single API call using a normal token.

[ranguna]

I get what your saying, but this is resonating with me and making me feel for the author:

Cursor: we have top notch safeguards for destructive operations, you have our guarantee, we are the best

Author: uses their tools expecting their guarantees to be true (I would expect them to have a confirmation before destructive operation outside their prompt, as a coded system guardrail)

Cursor AI: Does destructive operation without asking

Author: feels betrayed.

So yeah, I think the author is right because they trusted Cursor to have better system guardrails, they didn't (agents shouldn't be able to delete a volume without having a meta-guardrail outside the prompt). Now the author knows and so do we: even if companies say they have good guardrails, never trust them. If it's not your code, you have no guarantees.

[postexitus]

Sorry - still author's fault. They didn't understand how LLM's work. They thought Cursor implemented some magic "I control every action LLM takes" thing. It's impossible.

[laszlojamf]

right. But cursor _said_ they had some magic. At some point you have to trust vendors. I don't know exactly how AWS guarantees eleven nines of durability on S3. But I sure hope that they do.

[JohnMakin]

Here is what they say, at the very top they explain that llm's are inherently unreliable. It looks like they offer security tools and safeguards, but they also provide an auto run option. There is nothing a vendor can really be responsible for someone shooting themselves in the face. You can argue that they shouldn't provide that, but that's what people want, so they do, with warnings.

It sounds like this user either didn't use security controls, approved prompts they didn't understand, or disabled the checks entirely. Working in IT/tech a big chunk of my life so far and seeing all the dumb crap people who even know better do, I would bet my house on that being the most likely scenario rather than cursor somehow being at fault here.

https://cursor.com/docs/enterprise/llm-safety-and-controls

[PUSH_AX]

yeah and when you interview the junior dev who also convinces you they're smart and have something special, they also delete prod and guess what... not that devs fault.

[dpark]

> At some point you have to trust vendors.

You absolutely do not. When someone makes an unbelievable claim, such as having magic guardrails for LLMs that prevent dangerous actions (what would that even mean?!), you don’t have to trust that claim.

If you trust someone’s claim without justification, that’s on you.

[JohnMakin]

They don't even claim this:

https://cursor.com/docs/enterprise/llm-safety-and-controls

[dpark]

Yeah. It would be pretty dumb for them to make that kind of claim.

Thanks for providing that doc.

[wolrah]

> At some point you have to trust vendors. I don't know exactly how AWS guarantees eleven nines of durability on S3. But I sure hope that they do.

Trust is earned, it's built on reputations at the individual, corporate, and industry-wide levels. AWS has 20 years of reputation on which I can judge the value of their promises.

Not only has the LLM industry (it is not "AI" and never will be) absolutely not earned anything like that level of trust, the thing the technology has proven most effective at is in fact scamming. Making up something that looks/sounds convincing, especially if you aren't thinking too hard about it, is what they're best at. Combine that with a lot of money flying around and trust levels should be somewhere around "Elon Musk promises".

At this point there have been so many blatant examples of why you should never give a LLM "agent" control over production systems, but the allure of just giving some vague direction to a chatbot and telling it not to screw things up it just irresistible to some like Sideshow Bob stepping on rakes [1].

If everyone around you is whacking themselves in the face with the rake, and you know you can avoid it just by using your brain and not stepping on the rake, and avoid entirely by just keeping your rakes contained, but a rake vendor comes to you saying that instead they have built a new rake that they swear won't whack you in the face even if you leave it right in your walking path, do you trust them?

1: https://www.youtube.com/watch?v=ouau9SVVrBA

[fg137]

I mean, AWS doesn't really "guarantee" anything, they just say if they can't meet the bar they'll refund you in credits which is equivalent to money.

[ranguna]

Yeah I wasn't clear with "the author is right", I think they are right to be frustrated, but that doesn't clear their own fault in the matter It's just that it wasn't their fault alone.

This is not a polarizing issue, it's not just the authors fault, or cursors fault, or society's fault. It's everyone's, and we all got something to learn from this.

[ranguna]

Impossible?

You just have to add a human in the loop for destructive calls. Add an additional TOTP parameter to destructive calls that's generated from the agent UI that requires a human to click a button, which generates a code that's sent to the model and used in the call.

Why do you think this is impossible?

[postexitus]

Impossible without a human in the loop.

Having said that - even categorisation of destructive and non destructive calls is inherently not safe, unless you have very strict os level / VM like setup (everything read only, world access is through MCPs so it is not LLM deciding the destructive calls but the MCP etc. )

[manas96]

200% agree. If you decide to use this power you must accept the tiny risk and huge consequences of it going wrong. The article seems like it was written by AI, and quoting the agent's "confession" as some sort of gotcha just demonstrates the author does not really understand how it works...

[gwerbin]

The author definitely deserves a lot of blame here and clearly doesn't understand AI well enough to have a coherent opinion on AI safety.

But Railway bears some responsibility too because, at least of the author is to be believed, it looks like they provide no safety tools for users, regardless of whether they use AI or not. You should be able to generate scoped API tokens. That's just good practice. A human isn't likely to have made this particular mistake, but it doesn't seem out of the question either.

[dpark]

> You should be able to generate scoped API tokens. That's just good practice.

Fully agree, but given the rest of this story I don’t imagine the author would have scoped them unless Railway literally forced him to.

> A human isn't likely to have made this particular mistake, but it doesn't seem out of the question either.

The AI agent was deleting the volume used in the staging environment. It happened to also be the volume used in the production environment. 100% a human could have made this mistake.

[annoyingcyclist]

I kept reading and reading to find the part where the author took responsibility for any part of this, then I got to the end.

[computerdork]

I don’t know, software systems complicated, it’s pretty much impossible for one person to know every line of code and every system (especially the CEO or CTO). Yeah, it was probably one or two employees set this all up realizing the possibility of bad Cursor and Railway interactions.

if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t been given enough responsibility, or are just incredibly lucky.

… although, agreed, they were on the cutting edge, which is more risky and not the best decision.

[kokada]

There is a difference between making a mistake like this one and being humble (e.g., lessons learned, having a daily external backup of the database somewhere else, or maybe asking the agent to not run commands directly in production but write a script to be reviewed later, or anything similar) and just blaming the AI and the service provider and never admitting your mistake like this article is all about.

The fact that this seems to be written by AI makes it even more ironic.

[anonymars]

Indeed. I swear reality gets stranger and more implausible by the day.

"That isn't backups. That's a snapshot stored in the same place as the original — which provides resilience against zero failure modes that actually matter (volume corruption, accidental deletion, malicious action, infrastructure failure, the exact scenario we just lived through)."

[computerdork]

Agree in that this person seems to trying to shift blame, but still think he's right in that Cursor and Railway also have glaring weaknesses. Yeah, it's was somewhat of a perfect storm of mistakes with blame to go all around.

[dpark]

> Yeah, it was probably one or two employees set this all up realizing the possibility of bad Cursor and Railway interactions.

I’ve got a hunch the only person is the CEO.

The domain was registered in October 2025. The site has kind of a weird mix of stuff and a bunch of broken functionality. I think it’s one guy vibe coding a ton of stuff who managed to blow away his database.

> if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t been given enough responsibility, or are just incredibly lucky.

Mistakes are understandable. Having no introspection or self criticism, not so much.

[phatskat]

> if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t been given enough responsibility, or are just incredibly lucky.

I’ve definitely made bigger mistakes, but we also had an Oracle DB that could INSERT INTO…SELECT FROM -point in time- that pretty much put us back to the point before we started our migration. And of course we had backups rolling all the time, as well as our pre-migration backup. We had a good, competent team, and we overlooked a small but catastrophic detail - it can happen to anyone, the goal should be to have backups and failovers in place because things _will_ fail, at some point, and a contingency plan is just good practice.

[il-b]

If you can handle disaster& recovery, you shouldn’t be a CTO

[meisel]

Yeah the author really should’ve taken some responsibility here. It’s true that the services they used have issues, but there’s plenty of blame to direct to themself

[nzoschke]

And they decided to leave a token with destructive capabilities in the agents access, and decided to not have verified backups for their database.

My team practices "no blame" retros, that blame the tools and processes, not the individuals.

But the retro and remediations on this are all things the author needs to own, not Railway or Cursor.

- Revoke API tokens with excessive access

- Implement validated backup and restore procedures

- ...

[sombragris]

The whole use of AI agents in this context reminds me of the movie "War Games"

  > A strange game.
  > The only winning move is
  > not to play.

[reliablereason]

Right! Blaming an agent or anyone else is crazy. The author built a system that had the capability of deleing the prod database.

The system did delete the database cause the author built it like that.

[infecto]

Embarrassing post by leadership. I was surprised how quickly they immediately jumped into Railway and Cursor failures. I like living on the edge but I would never give an agent access to the prod DB.

[Zopieux]

It's hilarious how much they can't take any accountability for running a random text generator in prod, and they could not even be bothered to write their own tweet.

I do not feel sorry, but I do feel some real schadenfreude.

[rfwhyte]

They relied on frontier tech because GREED. Let's not kid ourselves that the decision to use AI here was done for any other reason than it would save this company the labor costs of actual human employees. They decided their profit was more important than the security of their customer's data, and now they are suffering the well deserved consequences for it.

[angrydev]

I love boring tech. It's reliable as hell and not as full of hidden surprises. Screw the cutting edge for serious work.

[estetlinus]

100%

Trying to run a blame game is such a facepalm.