[cratermoon]

I worked at a company with a self-inflicted wound related to retries.

At some point in the distant (internet time) past, a sales engineer, or the equivalent, had written a sample script to demonstrate basic uses of the API. As many of you quickly guessed, customers went on a copy/paste rampage and put this sample script into production.

The script went into a tight loop on failure, naively using a simple library that did not include any back-off or retry in the request. I'm not deeply familiar with how the company dealt with this situation. I am aware there was a complex load balancing system across distributed infrastructure, but also, just a lot of horsepower.

Lesson for anyone offering an API product: don't hand out example code with a self-own, because it will become someone's production code.