[nulld3v]

I tried solving this a different way for my selfhosted services.

Instead of running certbot on every server, I wrote a custom ACME client that runs on a master server and is responsible for requesting/renewing all certificates that I use. It also automatically deploys each cert to the correct server.

It is a single point of failure but it makes tracking certificate expiry, renewal and revocation so much easier.

[mholt]

Sounds like a fun project. Assuming you're talking about web servers, did you know Caddy can do that? Simply configure each one to use the same storage backend and Caddy will automatically coordinate management as a cluster, and share the certificate (and OCSP staple) resources.

(And depending on the storage backend, it's no longer a single point of failure. And even if storage is the failure, it's just storage, if it's down your servers will keep running.)

[nulld3v]

Huh that's actually quite interesting, I've never really looked into caddy as nginx has fulfilled most of my needs so far but I suppose it's about time I read up on it.

[GauntletWizard]

I wrote monitoring that was able to check all of my servers, all of my certificates, and alert me if certbot failed on any of them and their certificates were near expiry.

I call it "The Prometheus monitoring I already needed to make sure my servers are up and serving the websites they're supposed to"

[aargh_aargh]

That's what I pondered doing. Do you have any code you can open source? Thanks.

[nulld3v]

I'm not sure I want to publish it at the moment as I'm not satisfied with the quality of the codebase. It was just something I hacked together in 2 days lol.

The bulk of the work was already done for me as there was already an ACME library for my language.

I'll probably open source it once I have a chance to clean everything up.

[eatbitseveryday]

Published code provides more value than unpublished. Not cleaned up is most academic research code, for which there is no time to clean up.

And that’s okay.