[jrockway]

When your hosting company goes down, you're basically stuck waiting for them.

The only way to ensure that you never have downtime is to not use the Internet, and keep all your servers connected to a nuclear reactor, all inside a mountain. That covers power and not being flooded or hit by a tornado, anyway.

Even then you'd better be careful not to trip over any of the power cables or type "rm -rf /" as root.

[e1ven]

For mission critical things, you want to reduce the points of failure, and ensure that when things DO go wrong, you have a reasonable escalation path.

It's now been over 8 hours since it went down, and no fix from Google yet. 4+ days on the missing file.

If I was running in house, I could have entirely restored the mail server from tape by now. I could have swapped over to a hot-spare in a few minutes. I could have failed over to our backup internet service. I have a lot of options.

With Google, my option is to wait.. And hope my business doesn't lose too much money while Google gets around to fixing it.

[beambot]

As another example of handling cloud services going offline, consider this case by SmugMug:

http://don.blogs.smugmug.com/2007/01/30/amazon-s3-outages-sl...

"""So what are we doing differently? Simple. Amazon serves as “cold storage” where everyone’s valuable photos go to live in safety. Our own storage clusters are now “hot storage” for photos that need to be served up fast and furious to the millions of unique visitors we get every day. That’s a bit of an oversimplification of our architecture, as you can imagine, but it’s mostly accurate."""

You can always maintain a hot-backup, fail-over of your site on your own servers -- perhaps with reduced functionality until the scalable cloud services come back online. For a mission-critical site, this would seem to be a reasonable tradeoff.

[jlangenauer]

Are you, by chance, a Python programmer?

[jrockway]

English doesn't have """triple double quotes"""?

[cookiecaper]

Here's a guide to using Google Apps or any other outsourced email provider successfully:

1. Use a resilient external DNS system 2. Configure local mail clients to save messages for "offline browsing" or equivalent. 3. Use local mail client to check mail at least once a day

When email goes down, point the DNS to a pre-configured mail server. You have all your archives and shouldn't be missing any mail since delivery failures will try again not too much later.

As for the Google Docs issue, ensure that users are exporting vital docs frequently, preferably when they're done making a change. You should actually edit locally and use git. You can set it up to automatically upload changes to gdocs on push.

[jgarmon]

Okay e1ven, call this number 1.800.571.4984. Ask for Ryan. Or, if your prefer, email [e-mail] or tweet @ryanatbackupify. He's our inside sales guy. Backupify can't get your lost file back, but if it happens again, we can make sure its restored in minutes, not days (or never). 30-day trial of our premium product, for you or anybody else who reads this post. http://www.backupify.com/business/500

[jrockway]

People tend not to like accounts that only post to advertise one company.

[jgarmon]

Fair point. Sorry if I overstepped.