|
Roughly 10 years ago, I was working for a startup that offered a live conversational video service where you could also have hundreds (or eventually, thousands) of near-live watchers - with recording and later playback. The founder pitched the service to news orgs and celebrities. Anderson Cooper had a regular "show" there for a while, and we had a number of interviews with mostly 2nd-tier celebrities. When the service started, they made the decision to not actually delete any content (delete just set a flag which disabled the content but didn't actually remove it). Fast forward a year or so, and it became clear that a real delete was needed. So they had a junior engineer write up a sort of delayed sweep - delete all the videos with the delete flag set. But then, for some reason, they decided put the implementation behind a delay. Something like "actually delete all soft-deleted videos, but don't start doing it until 30 days from now". However, unbeknownst to the team, there was a bug in the implementation that deleted everything, regardless of whether the 'delete' flag was set. So one night, roughly a month later, all the content started disappearing from the site. One guy heroically tried to stop the process, but I think he was too late. The engineering director happened to be on a vacation down in South America somewhere and I think the founder fired him in a fit of pique. I managed to reclaim a small bit of content (basically the videos that were cached on the actual recording servers before they were uploaded to S3). You can imagine the technical over-reaction: * Delete switched back to a soft delete
* Turned on S3 object versioning
* Started redundantly copying content onto a totally different hosting service
This was fine (hah!) until we had to start taking down the inevitable child porn that always shows up on services like this - I got stuck with writing the takedown code and it took me forever to track down all the various tendrils of stuff.As you might expect, we lost a ton of users over mass content deletion and the service never really rebounded. The company held on for a couple more years, pivoting a couple of times, but eventually folded. |