[ChrisGreenHeur]

yep, it sounds like the first implementation step really should have been to gather a large test set of data and develop the system with that in mind after understanding the test data, starting with making tests from the test data.

[skeeter2020]

They explained the scenario though and it seems like a combination of rarer edge cases. It's great to think your awesome dev team and QA would have collected test data representing this ahead of time, but surely you've all been caught by this? I know I have; that's why we don't have flawless systems at launch.

[eszed]

OP said elsewhere that the specific discount that produced the $10M sales + refund combo was only active at five stores. That's the sort of edge case that you can't count on being in the training data. I'm as big a fan of real-data testing as anyone, but there's always something not represented.