Hacker News new | ask | show | jobs
by circusfly 800 days ago
kubectl rollout undo deployment/foo then replicate. The article predates kube though.

If it can't be replicated, loop back to the reporter and seek any clarifying details. If that fails, then communicate to management the extremely high risk of attempting a fix.
1 comments
yjftsjthsd-h 800 days ago
If the bug resulted from a rollout, you might be able to undo it. Or it might be a data problem, or it might be the result of factors out of your control.
link
circusfly 800 days ago
If the code in PROD was fine before the rollout, and there is a bug in the newly deployed code; then I was saying above to undo that deployment going back to the previous version then replicate the bug, preferably locally.

It seems like there a misunderstanding about what I wrote or there is a misunderstanding about how kube works and the misunderstanding isn't mine.

link
yjftsjthsd-h 800 days ago
> If the code in PROD was fine before the rollout, and there is a bug in the newly deployed code

Yes, if. My point is that that's a distinct subset of the possibilities, and you might not know whether it's true. Maybe you roll out a change at 10:00 and at 10:20 you start getting reports that certain key functionality isn't working. So you run your rollback... and now instead of 20% of users complaining, it's 100%, because that last update included a database migration. Okay, maybe you're very good about your data model so that doesn't happen, but you roll back... and nothing changes. It turns out that the problem was caused by a 3rd-party API you consume falling over. With the right guardrails, rolling back can often be an early option to improve the situation, but it's not a 100% cure-all.

link