[jermaustin1]

Same with Twilio. We have an internal server that does system alerts. We recently moved it to an IPv6 only host, and a few weeks went by and noticed there were no longer receiving alerts.

Turns out we could not connect to Twilio's API which is IPv4 only.

[tbrownaw]

So zero validation after that change?

[jermaustin1]

Couldn't tell you. I'm not part of the infrastructure team. I wasn't even aware the alerting service was moving.

QA found it a couple weeks later when they were testing alerting, and SMSes weren't coming through.

[vlovich123]

Zero observability and alerting too. Seems like they’re planning to be a productive future member of that team.

[jermaustin1]

Who? The infrastructure team that did the move didn't even tell anyone. They were decommissioning old servers, and moving the VMs to new hardware. I'm just a lowly developer that had to troubleshoot why SMSes stopped going out.

[vlovich123]

Observability and alerting is pretty standard devops. Both the dev and infrastructure teams dropped the ball here. But at least as part of remediating this you added alerting to make sure you’d notice when your twillio connection fails in the future, right?

[jermaustin1]

Even better, infrastructure enabled IPv6, and the issue was closed.

In corporate software development, we work the tickets assigned, and keep our KPIs up so that we don't face the wrath of the bean counters.