[amingilani]

Question: with a strict CI/CD in place, as well as a staging server, how can these problems be so common for Gitlab?

Isn't this exactly what CI is supposed to prevent?

Not blaming Gitlab for bad practices or anything, i'm just curious.

[koolba]

> Not blaming Gitlab for bad practices or anything, i'm just curious.

On the contrary, the backup snafu was caused by a series of bad practices. If that's how backups are handled I wouldn't be surprised if the rest of the testing infrastructure has issues as well. Heck, I'd be surprised if it didn't!

Particularly because a solid testing infrastructure works in tandem with your backup processes by restoring recent backups.

Nothing tests new code better than running it on a production restore and nothing validates backups better than using them on a regular basis for testing.

[michaelt]

Even with a staging server, things can pass testing but fail in production if the staging environment provides an imperfect simulation of the production environment - and that's almost inevitable.

For example, your staging environment servers should be connecting to a different database with a different password. If the password's right in the staging config but wrong (or missing) in the production config, things that work in staging can fail in production.

[justinlaster]

> things can pass testing but fail in production if the staging environment provides an imperfect simulation of the production environment

Your staging environment should match production, or it's not really staging at that point. It doesn't have to match it in _size_, just structure and process. Ignoring data loss, if you can't quickly switch staging to production it's not really staging. It's just a dorky test environment masquerading as a stage environment. It's also surprisingly not that difficult (the variation of difficulty depends on the type of data you're interacting with, and how isolated it needs to be) to "forward" a slice of real word traffic to your staging environment and monitor it for some duration of time.

>For example, your staging environment servers should be connecting to a different database with a different password.

Handled by proper CI/CD pipelines. Completely irrelevant to deploying new features, configuration for production specific users/passwords happens on the sysadmin/devops side of things.

[ajeet_dhaliwal]

Isn't this exactly what CI is supposed to prevent?

CI is only a facilitator, if their test coverage or quality isn't as good as it could be it won't make much difference. Also if it's due to load not sure how much loading testing they would do as part of CI. Having CI and writing automated tests is something everyone seems to agree in theory is a good idea but in my experience hardly anyone does it well because writing features always trumps writing tests. I am not talking about Gitlab specifically, I know absolutely nothing about their set up, only in general.

True story, I am involved with a startup that offers cloud based storage/reporting of test results (https://www.tesults.com) and my colleague just emailed the CTO of Gitlab yesterday to offer a promotion on a plan, very odd indeed to see this story on HN the next day!