[koollman]

external/independant backups are a thing. They would be part of a disaster recovery plan (and you should test the restore process and result from time to time, else they might be worthless when you actually need them).

I understand this is not helpful right now, but there are best practices and many ways to plan for this kind of stuff for good reasons. Some things can and will go wrong, and sometimes that may cost your business a lot, which means you have to anticipate (or accept) the possible costs and risks.

It is part of the job

[vorador]

This is very unhelpful. There's a difference between "someone fatfingered a command and brought this mysql machine down" and "our cloud provider shut down a core feature for our account".

When you design a new system do you plan for S3 to go down for more than a day? Do you have a fleet of offsite machines to smoothly transition to? If not, why not?

[koollman]

Well, it is about things to prepare before being in this situation, I acknowledged already that it was unhelpful now, since it is too late. It is advice for other business deciders looking at this, or for this one for changes to implement in the future.

Sometimes the plan is "well, this business had a good run, now it is over".

Sometimes it can be as cheap as "well, here's the documentation to run all my stuff elsewhere, starting from my external backups"

Depends on how much money you lose from a failure and how likely the failure is. As an additional point, you may not plan for s3 going down, but even a price hike on egress traffic may put a business in trouble if trying to move without already having external backups. So, as often, you have to do some cost/risk analysis. But having backups not controlled by your main provider is often considered a good idea.

Technical problems, human error, cost changes, various disagreements between provider and customer, can all be made easier if you have a plan B rather than being stuck with a single provider or solution

[misiek08]

If data on S3 is crucial for business to be alive? Of course! I would even think about writing data to 2 separate storages in real time to be able to switch easily. 3-2-1 backup is a no joke, especially when it's your business, not a hobby project. Having DR too. And it's not a legacy, 20 years old system, so it's much easier to do such plans and test them.

[vorador]

Do you run a multicloud system too? In case the AWS account gets shut down by billing issues?

[koollman]

You don't have to run it a entire duplicate of your current infrastructure, just plan for it to happen (depending on impact to your business). Then it means you can think about doing it after a few days of downtime, maybe earlier depending on difficulty/impact.

But, yes, you do have to plan to run it elsewhere and consider wether various features may make it harder for you to move on. vendor lock-in is a known business risk

[viraptor]

Running multicloud (at the same time) is for availability not disaster recovery. Just having another copy of the data outside is good for DR, even if you have to redeploy everything manually in a hurry.

[Brian_K_White]

"There's a difference between "someone fatfingered a command and brought this mysql machine down" and "our cloud provider shut down a core feature for our account"."

No there isn't. Those are both exactly the same problem with exactly the same solutions.