glad I wasn't the only one who had that as a first thought. as an ops guys I wasn't sure if I should be offended or just shake my head at the irony of it.
Well apparently someone didn't like my comment since it got down-voted. Whatever. As someone with a background in systems administration, it bothers me when this profession gets left out of the equation, far too often.
Sometimes i think Sys. Admin profession needs a messiah like NNT(for finance) with perhaps the outspokenness/attitude of Zed Shah. That way, there would be a whole lot of increased awareness for the challenges involved in Systems administration.
I would have to agree. But I would also have to agree that Heroku or AWS may have been a better choice than a 1&1 dedicated server in this particular instance.
However, Heroku and AWS are no substitute for good systems administration. They are good substitutes for data centers given signed and documented SLAs.
I mean that Systems Administration is as thoroughly exhausting a career path as programming. They didn't lose information because they coded something wrong or inadequately.
The most basic backup would be to have a remote backup of your files and databases. That's a simple shell script and cron job combination. I can understand that one may not get around to setting it up (I had a massive project and only started doing this after about 7 years of running it..) but it's definitely very important and doesn't need a ton of Sysadmin experience or setup time.
Are you guaranteeing that those files are in a good state when you ship them off? (i.e., flushed and synced to disk)
How about the DBs? Are you backing up the *SQL data files directly (and if so, are you guaranteeing that _those_ are in a good state?), or doing an SQL or other export?
Exactly which files do you need to backup -- user generated content, system configuration, logs, spool files? Do those files give you enough information to rebuild your machine(s) from bare metal? Or have you have documented your setup procedures?
Where are you storing these backups? And how many copies of them, and on what frequency, should you be keeping?
Are you testing your backups? Are you testing your recovery procedures? If not, how do you _know_ they're all working?
---
Sorry for the question dump, but there is a fair bit more to even basic backups than just syncing your filesystem to S3 / tarsnap / The Cloud TM every day.
Files are pretty straightforward, especially in a possibly low volume - startup that cannot afford a dedicated sysadmin stage, not going to address that here.
Simple database backup would be to replicate to a slave (just addressing mysql here for a simple case). Stop replication, backup, restart replication. Easy as that. More complicated scenarios need more complicated setups but hey, we're not talking about saving the last 0.0000001 seconds of data here, this is about losing all your data, from day 1.
Obviously I cannot comment on the specific backup needs for these guys since I don't know their app. However, I can still ensure that an app that is running on a shared hosting environment (meaning - it's not that complex..) has a reasonably good (24 hour snapshot ?) backup potential within a few minutes.
I think it'd have been a decent compromise if they even had a backup from yesterday with a few minutes/hours of effort upfront.
Not saying here that a sysadmin role isn't justified, just that there are a few steps you can follow so that while you don't have a sysadmin, nothing gruesome happens to you.
You're right and the cosmos has just proved it. I just went in to check my backups and found some that hadn't been running since Nov. 19th 2012. Well, lesson learnt :) - gotta test those backups!
Perhaps it's me, but I don't see the roles of SA and Dev as interchangeable. Yet, HN seems to present a world where all Devs are SAs. As a SA, it's frustrating.
All of that to say; I agree with what you're saying. I just wonder how to get the message across that maybe both roles aren't the same...
I think the point is when you're starting off you have to wear many hats, and since HN is a lot about start-ups, you'll see that often here. At a certain size/load I think most people on HN would recognize that you need specific talent sets (and they might not be interchangeable).
They aren't interchangeable, but there is a significant amount of overlap in the position, especially when you begin operating at scale. The amount and type of automation required when you're dealing with thousands of servers demands a breadth of knowledge that spans both domains.
They're synergistic roles, yes, and ideally you would like one with a mix of both [either a dev with sufficient experience in system administration or a system administrator with sufficient experience in development] to tackle the messes that crop up, but I haven't seen anyone claiming they're interchangeable - just that most of the population of HN is some overlap of the two.
This is likely because many of us are part of small startups where the roles are often interchangeable in the early days: the same small engineering team is responsible for both "writing code", backups, infrastructure, and everything else technical.
It's very difficult to have your foot in both worlds and remain current in either. I did for a while, but I'm more firmly planted in the dev world now. So I can say, with some confidence, that they aren't very interchangeable.