[RobSpectre]

I could have explained this better. Many usage events on the Twilio API generate a billing transaction (e.g. call, SMS message, phone number purchase, etc). When the transaction attempted to apply against a balance of zero and the customer had auto-recharge enabled, the billing system would trigger a charge attempt. With balances set to zero and read-only, each subsequent usage event would trigger a charge attempt, resulting in the erroneous charges.

[ahk]

So, the users most affected were some of your most active/high traffic ones? Ouch.

[patio11]

While it's unlikely that Appointment Reminder is one of Twilio's largest accounts, we do have a number of customers in the Eastern Standard time zone, and due to customer usage patterns we have a predictable spike in outgoing calls and SMS messages which happened to coincide with the early-morning PST Twilio problem.

Under normal circumstances, I have Twilio set up to bill us $500 if the balance ever dips below $500. That $500 is our rebilling increment. Each SMS message and phone call we made caused us to get charged our rebilling increment. We hit $3,500 before our credit card company started rejecting charges. I think that if they hadn't, we would likely have saturated our credit line.

I'm thrilled with Twilio's response to this issue, and most other transient issues I've had being a Twilio customer. The system is mostly rock-solid reliable. I actually went to bed during the middle of this event (midnight, Japan time) because a) my systems were reporting that our messages were successfully going out (so no customer-visible downtime) and b) I had total confidence in Twilio to take care of things. And they did.

[RobSpectre]

An incident like this is pretty painful for every customer affected. Hope they feel this explanation, the prompt refund of the erroneous charges and credit represents to them it is a pain we very much share.