Hacker News new | ask | show | jobs
by nonameiguess 509 days ago
The simplest things you can do are either:

- Stay within the cloud provider's ecosystem as much as possible, including for domain registration and DNS. All records then should be pointing to resources that include your account id in them and can't be taken over by others. If you delete the entire account, there'd be nothing to take over.

- Do everything with Infrastructure as Code, including DNS. If a single "terraform apply" creates everything, then a single "terraform destroy" deletes it all, leaving nothing dangling, provided of course that it is setup correctly and doesn't error out midway through a run.

Otherwise, it's a matter of being thorough. Automate what you can, including creating and deleting resources, if not through a single cloud provider API or some standard IaC product, then roll your own software to do it, but have software do it. Regularly roll out and tear down entire test installations of full systems, including valid DNS records. When you intend for them to be gone, ensure they are really, truly gone.

If you can't automate it, then yeah, checklists.

It's one of those things that is simple but not easy. It takes an organization that respects the tedious and time-consuming nature of ops, plans for it, and doesn't push people to cut corners for the sake of speed when the first time trying to do something takes much longer than someone's uninformed first guesstimate.

Really, automate. At a small enough scale, it doesn't matter, but if you're Mastercard doing this kind of thing thousands of times over the course of decades, humans will inevitably make mistakes. Software will make mistakes, too, but at least when you test software, it will do the same thing every time it is tested. Humans do not provide that guarantee, even if they have checklists.

Edit: Note the above is not true for LLMs, so when I say use software, I mean classical deterministic software. Don't have AI do it for you, because LLMs can and will produce different responses every time you make the same request. Don't devolve to making software that is just as flaky as humans.

1 comments

> Stay within the cloud provider's ecosystem as much as possible, including for domain registration and DNS

Alas, if you follow this advice to mitigate this particular risk, you're completely hosed if your cloud account gets taken down or compromised. Which is why the standard advice is to do exactly the opposite and make sure your domains and DNS are separate from your cloud provider.

What if you have your domain registered outside of your cloud provider, but have your nameserver on your cloud provider's infra.

You can have another cloud platform configured with a duplicate nameserver, then go to your registrar and change the nameserver for your domain.Your replacement nameserver would then control any subdomain provisioning.

I think that would deal with the risk somewhat, though could be missing something.

> your cloud account gets taken down or compromised

In risk assessment this risk should be resolved as „avoid“, because loosing DNS will be the secondary concern. Data is even more important. I agree that domains should be registered elsewhere and it’s good idea to have the backup of the zone.