| HN Mirror

Y	Hacker News new \| ask \| show \| jobs


	by Morgawr 4276 days ago
	>Backups are on another server in the same datacenter. You might want to look into moving backups to an offsite location. Datacenter administration 101, if there's a fire or an earthquake or any other natural disaster, you definitely don't want to lose all your data because it was all in the same place. Just a "different machine" doesn't cut it.

1 comments

lsc 4276 days ago

>You might want to look into moving backups to an offsite location. Datacenter administration 101, if there's a fire or an earthquake or any other natural disaster, you definitely don't want to lose all your data because it was all in the same place. Just a "different machine" doesn't cut it.

You know how often earthquakes happen? fires?

You should guard against the common things first. Sysadmin error is a way more common way to lose data than earthquakes or fires. hostile compromise is also a lot more common.

Having the data on another server (rather than in a snapshot) is a good first step because it protects you from RAID screwups, and sysadmin error in whatever snapshot layer you use.

Generally speaking a reasonable defense against compromise is a reasonable defense against sysadmin error... best practices (and I know of no vps provider, including myself, that actually adheres to these best practices.) are to set up your backups so that the production root can not overwrite or delete old backups. Ideally, no one employee has write access to both production and backups, that way no one person, even if their credentials are compromised by a hostile, or even if they become hostile, can wipe all your data. (note, this requires an off-site backup. Physical access is write access, but this really only protects you from an employee who is willing to risk jail time to hurt you, and while that happens, it's pretty rare compared to an employee's login credentials being compromised. I would setup protection so that no one employee can overwrite both production and backup remotely before spending the effort and money to haul all my backups to another location.)

But like I said, as far as I know, nobody actually does that (and its a difficult sort of thing to verify.) - for the low end VPS market? if they have backups on another server (rather than in a snapshot or something on the same server) they are doing okay.

But really... if you care? you should do your own backups. As a customer, you don't have any choice about letting your hosting provider have write-access to production. Make sure you have a backup somewhere that they don't have access.

link

Morgawr 4275 days ago

>You know how often earthquakes happen? fires?

Yes, let's save money from buying fire extinguishers, that doesn't happen very often anyway. I mean, for big companies with money it's okay to have safety procedures but for small companies we can expect them to ignore it, right? After all "nobody does that anyway" :)

Sarcasm aside, such accidents can wipe out your entire business and destroy your credibility, just because something doesn't happen often it doesn't mean you should ignore it. Plus, depending on the area you live in, earthquakes can be very common, but I digress.

All of this aside, I agree with the rest of what you said, but it doesn't invalidate the fact that if you want to have proper data safety practices you need offsite backups (I'm talking as a service provider, not as a user. Of course you want to backup your data yourself too, as a user)

link

lsc 4275 days ago

Nobody is saying that off-site backups are a bad thing. they are certainly part of a complete DR setup.

And yes, all of us know that if we lose the data? we lose the customer, and worse, we lose the customer and they say (justifiable) bad things about us. If you lose all your data? yeah, you are out of business.

Even so, there's a huge difference between how a low-end hosting company is run and how an "enterprise" datacenter is run, and expecting enterprise reliability at low-end pricing is... not realistic.

If you think that your low-end VPS provider is doing everything possible to back up your data... you are likely to be disappointed. Hell, I don't have regular backups, or offer for-pay backups at all myself. I'm working on it for the next version of my management software, but for now, I'm very up-front with my customers that they need to back up their own stuff, and my architecture is such that most of the time, data loss is confined to one server, and yeah, if someone got in and wiped it all out, I would very clearly be bankrupt. but... I'm only saying this because yeah, in this market, you don't get "enterprise level" backups. If you want 'enterprise backups' you have to do it yourself, or pay 'enterprise money'

You can say that it's like not having a fire extenguisher, and I'm not going to argue with you, but I've worked in this sector for well over a decade, and yeah, that's just how things are done. A low-end hosting setup is going to be way different (and way cheaper) than an "enterprise" hosting setup.

and I do know that many of my competitors have gone out of business because both backup and production could be written to by the same user (do you remember HyperVM? it was a disaster for many in the industry.)

I don't know of any competitors that have gone out of business because of physical destruction of their datacenter.

just saying... off site backups are good... but I would get the security and 'defense in depth' setup squared away first, and nobody does, because it's not on the checklists.