[Kiboneu]

Nah, you can snapshot every 15 minutes. The snapshot interval depends on the frequency of changes and their capacity, but it's up to them how to allocate these capacities... but it's definitely doable and there are real reasons for doing so. You can collapse deltas between snapshots after some time to make them last longer. I'd be surprised if they don't do that.

As an aside, snapshotting would have prevented a good deal of horror stories shared by people who give AI access to the FS. Well, as long as you don't give it root.......

[john_strinlai]

>Nah, you can snapshot every 15 minutes.

obviously you can. but, what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot? that is what matters.

in any case, the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all.

[Kiboneu]

> the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all.

I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin. (Did you read the OP?) That's a /you/ problem if you rely on infrequent backups, especially for a service with so much flux.

> what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot?

? Why would I know what their internal operations are?

[john_strinlai]

>I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin.

>Why would I know what their internal operations are?

i mean... you must, right? you know that once-a-day snapshots is not relevant to this specific incident. you know that their sysadmins are apparently competent. i just assumed you must have some sort of insider information to be so confident.

[Kiboneu]

I think you are misreading my comments and made a bad assumption. The reason I'm confident is because this has been my bread and butter for a decade.

[john_strinlai]

>The reason I'm confident is because this has been my bread and butter for a decade.

my decade of dealing with incompetent sysadmins and broken backups (if they even exist) has given me the opposite of confidence.

but im glad you have had a different experience

[Kiboneu]

> my decade of dealing with incompetent sysadmins and broken backups (if they even exist) has given me the opposite of confidence.

Oh, I agree that the average bar is low. That's part of the reason I do it all myself.

The heuristic with wikimedia is that they've been running a PHP service that accepts and stores (anonymous) input for 25 years. The longetivity with the risk exposure that they have are indicators that they know what they are doing, and I'm sure they've learned from recovering all sorts of failures over the years.

Look at how quickly it was brought back up in this instance!

So, yeah. I don't think initial hypothetical counterpoint holds water, and that's what I have been pointing out.

[sobjornstad]

Nowadays I refuse to do any serious work that isn't in source control anywhere besides my NAS that takes copy-on-write snapshots every 15 minutes. It has saved my butt more times than I can count.

[Kiboneu]

Yeah same here. Earlier I had a sync error that corrupted my .git, somehow. no problem; I go back 15 minutes and copy the working version.

Feels good to pat oneself in the back. Mine is sore, though. My E&O/cyber insurance likes me.

[gchamonlive]

The problem isn't the granularity of the backup but since the worm silently nukes pages, it's virtually impossible to reconcile the state before the attack and the current state, so you have to just forfeit any changes made since then and ask the contributors to do the leg work of reapplying the correct changes

[Kiboneu]

Why would nuked pages matter? Snapshots capture everything and are not part of wikimedia software.

[gchamonlive]

The nuke might be legitimate?

[wizzwizz4]

That's not a lot of state lost. Destructive operations are easier to replay than constructive ones.

[gchamonlive]

Is Wikimedia overreacting then?

[wizzwizz4]

No: from what I can tell, they're being conservative, which is appropriate here. Once you've pushed the "stop bad things happening" button, there's no need to rush.