Cert-manager has great support for a number of providers[0] including AWS, CloudFlare, Google Cloud, and Azure.
I recommend this not just for internal IP setups, for actually for all setups, since DNS verification is more robust than HTTP verification, particularly if you have issues with load balancers, or if Let's Encrypt decides to deprecate a protocol again [1].
I like DNS challenges, but I don't see how it matters for deprecation of an ACME challenge type. The dns-01 challenge could just as easily for some reason need to be deprecated.
The two likely reasons for such deprecation would apply just as well:
1. Updated Baseline Requirements or a programme policy requirement at any of the major root trust stores could forbid this challenge or require it to be substantially modified, obsoleting it in its current form.
2. The BRs don't change but Let's Encrypt finds they need to adjust this particular implementation in a non-compatible way so they deprecate the current challenge.
It can be easier to do DNS challenges, but it can also be very rough, depending on all the moving parts in your system.
Both of those are reasonable concerns, if all other factors were ignored.
However, in practice, the DNS challenge (which demonstrates control over DNS) is greatly preferred over HTTP/TLS challenges (which demonstrate control over a single port).
DNS is likely to be the only way to get a wildcard certificate, and HTTP/TLS will likely end up further restricted once SRVNames in certificates can be gracefully rolled out.
As such, deploying the DNS based control is absolutely the best thing to do, and HTTP/TLS should be seen as legacy-compat fallbacks that may become more difficult in time. Either certificates become less scoped than “entire domain” (as they are today with dNSName SANs) or it becomes more difficult to use a single port to prove authorization for an entire domain.
Any HTTP/ALPN request first begins with DNS, so if you’re trying to compare those, they all share the same base issue. In theory, this can be mitigated by DNSSEC, but that’s not relevant when comparing these validation methods.
However, both the HTTP and ALPN methods only demonstrate control over a single port (or .well-known resource), while the DNS method demonstrates the full ability to alter any/all names.
Verification via DNS is not without issues. If you have more then one DNS server the verification record need to propagate to all servers. If you for example use anycast DNS you will run into issues. Letsencrypt uses Google name servers for lookup which is problematic because they do not behave, they will for example not try secondary dns servers if the first try fail, making the Letsencrypt verification also fail. And because of these issues and if you have many domains you will quickly reach Letsencrypt quota.
> If you for example use anycast DNS you will run into issues.
You're not wrong, but this assumes that you use the your 'service hostname' for verification as well, rather than using CNAMEs.
So let us say you want to have "svc1.example.com" in your cert: you could put the ACME challenge under there, but if you have anycast delays that's a problem (as you mention). (A kludge is putting a 'sleep' somewhere to allow for propagation.)
So instead what you can do is have "_acme-challenge.svc1.example.com" be a CNAME that points to (say) "_acme-challenge.svc1.dnsauth.example.com". This sub-domain is not anycast, and may actually be a single machine that is used solely for this purpose.
The LE/ACME server goes to your main domain, finds a CNAME, and follows that to the real record and verification is achieved:
The CNAME has to be set up initially, but can be left lying around otherwise.
This is how $WORK deals with getting LE certs for internal domains: we create a CNAME record (but no A records) for the internal hostname in our external DNS that point to our "dnsauth" domain which gets updating by internal clients via an API.
Oh, this is an interesting trick... I think I'll need to investigate further.
Do you use a custom acme/dns updater for automatic renewals?
[ed: ie - if I understand correctly, I could point:
_acme-challenge.example.com via CNAME to auth.other.example.net - but then I'd like a command to check renew my example.com certs - and it would ideally use an api/dns update to manipulate the auth.other.example.net TXT (or CNAME to something like a15ce5b2-f170-4c91-97bf-09a5764a88f6.auth.acme-dns.io) record when I ask for a check/update of '*.example.com' certificate.
As far as I'm aware, most tooling assumes that you can/will (programmatically or manually) update the _acme-challenge.example.com record directly when issuing/updating an example.com certificate?]
First we use standard LE/ACME clients: either certbot or dehydrated. They ask for something like svc1.int.example.com ($DOMAIN).
In the hook script(s)† we manipulate the $DOMAIN string to put it into dnsauth.example.com ($AUTH_ZONE) sub-domain and send that new string to the DNS server that handles the dnsauth zone (and only that).
Before all of this we would have set up, in our public-external DNS, a CNAME record to point svc1.int to svc1.int.dnsauth.
The ACME client only thinks about $DOMAIN and the cert-issuing LE server only thinks about $DOMAIN. But the "in between" does not: by doing text manipulation (expr(1) is handy in shell scripts), and DNS redirects, the "in between" uses not-$DOMAIN for verification, but rather TXT records in $AUTH_ZONE.
All with standard ACME clients and some jiggery pokery.
We ended up creating some custom scripts called via SSH, but there are (now) DNS servers written specifically to handle REST API calls [0] and one can use lexicon [1] for just about any commercial DNS service.
Ok, thank you for the details and managing expectations. This still seems to warrant some experimentation.
In my case I'm mostly interested in delegating a domain/sub-domain somewhere I can easily update (be that run my own dns, host it somewhere with an api) - while having my main domains on a more boring/static dns infrastructure - yet still easily get certs for things like imap.example.com - which would not run a web server. And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.
> yet still easily get certs for things like imap.example.com - which would not run a web server.
Well, depending on the OS, you could start up a web server during the LE verification process and then bring it down once that's done. You'd only have to run in on port 80 for probably less than a minute.
But yes, you could this mechanism to have "_acme-challenge.imap.example.com" (which is what the ACME protocol uses) be a CNAME to point to something.auth.example.com that is more dynamic. Or even a completely different domain like foo.bar.example.ORG.
In your example.com zone file you'd put NS and A(AAA) records to point to the DNS server that handles the queries for the auth sub-domain.
> And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.
It's easier to run the ACME client on the host in question, and I'm not sure what it gains you to have it run somewhere else. That being said, there are ACME clients with a bit of a focus on being run 'remotely' from where the certs actually live:
Where have you seen Let’s Encrypt using Google’s servers?
CAs are required to run full recursive resolvers, up to the root, and can’t just point at someone else’s DNS infrastructure. Which, if you think about it, is what you want: you don’t want the CA just trusting someone else is being honest, you want to go to the authoritative source.
i had this issue but i just set the time to wait for propagation high enough to be somewhat certain and had no further issues since (its 10m i think). it does not really matter to me how long it takes as its an automated process... should be finished before expiration though ^^
having multiple nameservers is pretty standard and often mandatory requirement set by the NICs. Its just that my secondary is sometimes not fast enough to transfer the zone after a change notification which triggers this issue.
Also, retrying after an authoritative nameserver said there is defacto no record seems pretty wrong to me... i doubt they use google DNS or anything really. in order to avoid caching issues they very likely resolve names recursively without (the usual) caching
In which case it may be advisable to delegate the acme-challenge record to a different DNS provider, if doing so allows you to sidestep the anycast issues.
The two likely reasons for such deprecation would apply just as well:
1. Updated Baseline Requirements or a programme policy requirement at any of the major root trust stores could forbid this challenge or require it to be substantially modified, obsoleting it in its current form.
2. The BRs don't change but Let's Encrypt finds they need to adjust this particular implementation in a non-compatible way so they deprecate the current challenge.
It can be easier to do DNS challenges, but it can also be very rough, depending on all the moving parts in your system.