[bluedino]

We had a 'microservice' running on a server that had a bunch of other random things on it, it ran for 4-5 years until the server got decommissioned.

Nobody knew what all ran on that server, worse yet nobody knew that particular service ran on it. The person who wrote it was long gone.

It took a day to troubleshoot, a day to figure out what actually happened, and 5 days to get the server backup and running.

A couple months later, someone shut the server down again. It only took three days to fix it the second time.

In order to ensure this would never happen again, there were about 15 meetings, 20 people were involved, and then service was re-written and hosted on Azure (with the rest of some of our stuff). It's probably failed about 100 times since then, in about a hundred different ways.

This thing was maybe 500 lines of php.