[jnw2]

If the production server is running as a VM to begin with, how hard is it to take a snapshot of the whole VM and replicate that? (It does sort of break down if you have a database that's continuously updated and you want to try to have a continuously updated copy of the database, though.)

[gaius]

How hard is it? Well, about as hard as writing bug free code is. You don't need testers, right, just don't put bugs in in the first place! Or is it not easy after all?

[lsc]

well, just taking an image and throwing it somewhere is pretty easy (takes a bit of disk/network bandwidth, but it's not hard, if you have enough of those)

If you are short on disk/network bandwidth, the 'snapshots over the network' capabilities of things like zfs can help, while still remaining fairly simple.

hell, plain old rsync can work pretty well, and helps a lot with network bandwidth, even if it doesn't help so much with disk bandwidth.

And yeah, just taking disk images (or even tarballs) and stashing them somewhere cheap is a good start. much better than nothing.

Now, the downside to untested backups? when you need the data, something always goes wrong and it becomes this huge emergency that you have to deal with right now - and sometimes? even if you do everything right from that point on, the data is just gone.

I mean, that's a fair tradeoff a lot of times; if everything goes well, you've put little effort into backups, but you are still at a point where if something goes wrong, you can work hard and probably get your data back. I mean, yeah, you still have a reasonably high chance of losing everything, but your chances of recovery are way better than if you have no backups, and the effort level (if you don't need the backups) is not that much higher than no backups.

The thing is, the original subject was people who didn't want to deal with backups. "Oh," I thought, "Why don't you pay me to worry about your backups?" If I'm going to accept money from you to manage your backups? well, I have to set some expectations. most people aren't going to want to pay very much for untested backups, 'cause it really isn't that much work, until the excrement intersects the turbines, at which point it's a giant emergency, and a huge amount of work. At that point, I'm in a position where some $10/month customer is expecting me to do a couple grand worth of work; I mean, as a $10/month customer, I have never seriously looked at your app before. I've gotta restore from a backup that was taken without thinking about how your app works and make it work when I don't even know what "working" looks like, so it's going to be a lot of effort. Either I do it for free and I'm unhappy, or I hit them with a huge unexpected bill, and they are unhappy. (well, a $10/month customer, generally speaking, simply isn't going to pay a multi-kilobuck bill, and this is easily several thousand dollars worth of work, so in that case I'm unhappy and they are unhappy.) I try to be careful about setting expectations that I can actually meet, and I try to only accept customers who will be happy with a service that I can actually provide. - So yeah, while I think untested backups are a fine first step, and might even be what makes most economic sense for most people, configuring untested backups is not a service I would be willing to provide. If that's what you want, do it yourself.

However, if I put more effort into a tested recovery procedure up front, and more effort into testing that procedure in a routine, verifiable way ongoing? Sure, a lot more effort is being spent, effort that didn't need to be spent if nothing goes wrong, and I'm going to charge you for that. It will be more expensive. But I'm charging you a predictable amount; you knew what you were getting into when you signed up, so you aren't so unhappy. Then, in the unlikely event of a water landing? It's really not that big of a deal; you've been paying your monthly dues, I know the basics of your system, I've set aside some spare hosting resources, and I feel okay about getting woken up to make sure your restore goes okay, and the restore probably will go okay. I mean, it'd cost more money up-front, but it would set expectations in such a way that those expectations could be met.

My break is over; I need to get back to dealing with one of those giant emergency situations caused by a lack of tested backups... on one of my systems, in fact.