[msiemens]

I brought down production for a project — by running the deployment CI pipeline for already deployed commit. A couple of minutes later, production was thoroughly dead.

Turned out that my coworker had set up the CI process to use a PHP-based zero downtime deployment scheme where each release was deployed into a folder with the commit hash as name and then a symlink was updated to point the web root to this new release folder.

But, critically, he also configured CI to delete old releases at the end of the deployment pipeline - by removing all release folders older than three days. And by re-deploying a commit older than three days, after uploading the code and updating the symlink, the release‘s folder was considered old and deleted at the end of the pipeline, leaving the webserver with an empty directory as web root.