[isclever]

My takeaway:

1. Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working.

2. Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account.

3. Be able to jump ship easily, have backups of your zone, already know where you will transfer to.

[djsumdog]

> UptimeRobot and Hetrixtools are good starters with generous free tier

Are there any open source status pages/monitor programs that have build-in checks for HTTPS, DNS records (ipv4/6), arbitrary port checks, etc? I'd rather just setup a status page/alert app on a $5 minimal DO/Vultr node and self-host/support/contribute to a FOSS program than use a commercial provider.

[stevekemp]

I wrote a scalable system for this:

https://github.com/skx/overseer/

Handles SSL-checks, DNS-checks, SMTP-checks, & etc. Runs a thousand-checks every two minutes for me, give or take. Pluggable output via a redis-queue.

[isclever]

Here is a good list: https://github.com/n1trux/awesome-sysadmin#monitoring

Maybe one fits what you are looking for.

[falcolas]

<opinion class="unpopular">

Nagios. Or its descendant with a better configuration language, Icinga2. They're fairly easy to do a minimal install and configure in a container or on a VM.

</opinion>

[vorpalhex]

You need to host across several nodes in different geographic locations and data centers to resist network splits. Then you need some way to slowly roll out upgrades to your monitoring platform over time.

[djsumdog]

I'm just talking about my personal infrastructure. If I host my crap in Vultr or Linode, I should be able to buy one cheap node on another provider just to run a simple status app: something with celary or sidekiq jobs to check my other stuff and intervals and generate a page with some red/yellow/green dots.

[vorpalhex]

How do you know if the monitoring node goes down at the same time as the other servers?

Remember that Linode/Vultr/etc don't run their own datacenters, they share datacenters and sometimes downtime events can exist outside of datacenters.

[iudqnolq]

If you want email or text message alerts I would assume that's a complicated enough system you would want uptime alerts on it, and so on recursively ad infinitum.

[unilynx]

If you can set up nagios (which one would probably consider an interesting evening challenge if you were already willing to go for your own monitoring droplet) setting up pushover or amazon sns (for sms) should be easy enough.

[falcolas]

FWIW, a lot of cellular providers have an email gateway for delivering SMS messages. There's also paid SMS gateways, and options for providing arbitrary push notifications to smartphones.

[iudqnolq]

I'm pretty sure the free email gateways have no posted SLA. Plus, that requires a reliable email server, which would also need its own monitoring.

[falcolas]

This was a few (3) years past, but they accepted root@localhost sendmail messages just fine in most cases, and delivered alerts within a minute or two of sending. We didn't rely on this long term, but it was a "good enough" first pass.

I'd probably recommend using one of the gateways (or a more fully-featured service like Pagerduty) for more serious businesses, but for personal use (or where an outage detected the next day isn't crippling), it's remarkably useful.

[iudqnolq]

I would try to set up a completely open source monitoring setup just for fun, but once I'm paying for SNS I personally would rather just pay epsilon more and buy/rent the whole system. I get that may just be personal taste. I absolutely don't trust myself to run my own highly-reliable mailserver to send status alerts.

[petre]

Just send them locally and pull them with IMAP onto your phone.

[iudqnolq]

I didn't even know that was possible. Thanks for teaching me something new. It's always nice to learn I need less SAAS magic than I thought.

[hedsht]

check out https://github.com/hunterlong/statping - thats what i'm using.

[iudqnolq]

> Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working.

Lesson learned :)

> Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account.

I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money.

> Be able to jump ship easily, have backups of your zone,

Luckily I have that

> already know where you will transfer to.

Any suggestions? Ironically I recently moved from Google Domains to Cloudflare because I was worried about issues with opaque support. I've learned my lesson picking based on cost alone, but I'm a college student who can't afford something too heavy-duty.

[woofcat]

>I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money.

Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare.

[iudqnolq]

> Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare.

I believe it was both.

If I have a registrar outage I'm hosed. If I don't have a registrar outage and do have a DNS outage I can recover with a little work. But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well?

[petre]

> But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well?

Because they have just proved being uncapable of doing it? Because redundancy? Because you shouldn't keep all your eggs in the same basket.

I've been self hosting for at least 15 years and did not have any huge problems like the domain becoming non resolvable. I would never host my DNS on my registrar's infrastructure. It's being sloppy and lazy and it gets you embarassed.

[isclever]

A domain registered at a provider (but not DNS) can be down with no impact to your domain, so long as the domain is still in the TLD root servers, everything will keep going.

[iudqnolq]

Thank you for teaching me something new. I didn't know that got cached.