[daxfohl]

I am a devops engineer and understand your point. But there's a huge difference: legacy code doesn't change. Yeah occasionally something weird will happen and you've got to dig into it, but it's pretty rare, and usually something like an expired certificate, not a logic bug.

What we're entering, if this comes to fruition, is a whole new era where massive amounts of code changes that engineers are vaguely familiar with are going to be deployed at a much faster pace than anything we've ever seen before. That's a whole different ballgame than the management of a few legacy services.

[cagenut]

after a decade of follow-the-sun deployments by php contractors from vietnam to costa rica where our only qa was keeping an eye on the 500s graph, ai can't scare me.

[daxfohl]

That's actually a good comparison. Though even then, I imagine you at least have the ability to get on the phone and ask what they just did. Whereas LLM would just be like, "IDK, that was my twin brother. I'd ask him directly, but unfortunately he has been garbage collected. It was very sad. Would you like a cookie?"

I wonder if there's any value in some system that preserves the chat context of a coding agent and tags the commits with a reference to it, until the feature has been sufficiently battle tested. That way you can bring them back from the dead and interrogate them for insight if something goes wrong. Probably no more useful than just having a fresh agent look at the diff in most cases, but I can certainly imagine scenarios where it's like "Oh, duh, I meant to do X but looks like I accidentally did Y instead! Here's a fix." way faster than figuring it out from scratch. Especially if that whole process can be automated and fast, worst case you just waste a few tokens.

I'm genuinely curious though if there's anything you learned from those experiences that could be applied to agent driven dev processes too.

[cagenut]

it was basically a mindless loop, very prime for being agent driven:

  - observe error rate uptick
  - maybe dig in with apm tooling
  - read actual error messages
  - compare what apm and logs said to last commit/deploy
  - if they look even tangentially related deploy the previous commit (aka revert)
  - if its still not fixed do a "debug push", basically stuff a bunch of print statements (or you can do better) around the problem to get more info

I won't say that solves every case but definitely 90% of them.

I think your point about preserving some amount of intent/context is good, but also like what are most of us doing with agents if not "loop on error message until it goes away".