[OJFord]

Er, can we expect more information to follow?

1. How was the employee's account accessed? No 2FA?

2. Do employees ordinarily have access to customer secrets (e.g. API keys) or was there some further exploit?

3. The advice in OP for affected customers is to roll keys and SMTP logins. Couldn't/shouldn't you do that for them? Surely security should trump up-time/deliverability?

[ufmace]

For 3. I'd say no way. There's no way for Mailgun to know what services are doing with those keys, how important those services are to their customers, how difficult it is for the service owners to rotate their keys, and how much bandwidth they have to do that right now.

In an ideal world, every customer would have a good setup where they can rotate third-party supplier API keys painlessly and have plenty of bandwidth to handle security emergencies. Alas, there's a lot of bad setups out there, and some of them are critical to their customers' operations.

Nothing I've personally worked with had a setup bad enough to make that painful, but I'd be very worried about how reckless a service is to rotate API keys that aren't being actively exploited to do something dangerous without getting a positive confirmation from the customer.

[somedickhead]

All Rackspace employees are issued hardware or software RSA tokens and a VPN client.

I seriously suspect this was the job of an insider, not a compromised employee laptop.

[ralphm]

MailGun has been spun out of Rackspace almost a year ago.