[mikeash]

That is a crazy-ass quote. “We believe that no data has been lost... well, except for the data we keep. But you weren’t actually relying on us to save any data, right?”

I know, back up everything at least twice. But still, when somebody loses one of your copies, they don’t get to say “it’s cool, no data was lost, you have other copies, right?”

[gitlab-security]

To provide additional context, on GitLab.com, we maintain two weeks' backups. The last time we restored a single project repo, it was a significant effort that utilized many hours of an SRE's time to complete.

[mikeash]

What’s crazy isn’t the loss of data (that happens, and I don’t really expect most cloud services to be save me from explicit deletion requests) but rather framing it as no data being lost because your customers have it elsewhere.

[insomniacity]

"many hours" is too high. Regardless of this incident, if backups take too long to restore, you may as well not have them.

[GL-pharrison]

I agree, the longer it takes to recover the less valuable the backups are. However, in this context where we're restoring individual repo's (and all the metadata baggage involved) it's much different than a DB restore or other disaster recovery/prevention mechanisms.

The per-repo restore time today is not where it should be. We're working to speed this up so we can help users recover and get back to a productive state quickly.

[zifnab06]

Does support have the ability to restore keeparound refs? Internally (at least as of late 10 series), anything show in the UI (ie, merge requests) is also copied to refs/keeparound/sha1, which isn't presented to users.