[hinkley]

There were decidedly shitty engineering decisions behind that dumpster fire.

The biggest being that the only safe way to recycle feature flag names is to put ample time separation between the last use of the previous meaning for the flag and the first application of the new use. They did not. If they had, they would have noticed that one server was not getting redeployed properly in the time gap between the two uses.

They also did not do a full rollback. They rolled back the code but not the toggles, which ignited the fire.

These are rookie mistakes. If you want to argue they are journeyman mistakes, I won’t fight you too much, but they absolutely demonstrate a lack of mastery of the problem domain. And when millions of dollars change hands per minute you’d better not be Faking it Til You Make It.

[datadrivenangel]

The powerpeg feature flag had been deprecated for 9 years? In aggregate, yes, the engineering led to the disaster, but the specific engineers had been making everyone piles of money for a long time. The mistakes were fatal, but in a tiny amount of time on a system that old it's honestly surprising it didn't happen sooner.

https://specbranch.com/posts/knight-capital/

[hinkley]

Deprecated is not removed. I’m talking about airgapping EOL and resurrection of a feature flag code.

The fact that they hordes old code for NINE YEARS evaporates most of the remaining sympathy I had for them. Witaf.