[hedora]

I’ve noticed a strong tendency for older systems to accumulate “spaghetti architecture”, where newer employees add new subsystems and tenured employees are blind to the early design mistakes they made. For instance, in this system, it sound like they added a complicated health check mechanism at some point to bounce faulty nodes.

Now, they don’t know how it behaves, so they’re afraid to take corrective actions in production.

They built that before ensuring that they logged the result of each failed system call. The prioritization seems odd, but most places look at logging as a cost center, and the work of improving it as drudgery, even though it’s far more important than shiny things like automatic response to failures, and also takes a person with more experience to do properly.