[z3t4]

Verification via DNS is not without issues. If you have more then one DNS server the verification record need to propagate to all servers. If you for example use anycast DNS you will run into issues. Letsencrypt uses Google name servers for lookup which is problematic because they do not behave, they will for example not try secondary dns servers if the first try fail, making the Letsencrypt verification also fail. And because of these issues and if you have many domains you will quickly reach Letsencrypt quota.

[throw0101a]

> If you for example use anycast DNS you will run into issues.

You're not wrong, but this assumes that you use the your 'service hostname' for verification as well, rather than using CNAMEs.

So let us say you want to have "svc1.example.com" in your cert: you could put the ACME challenge under there, but if you have anycast delays that's a problem (as you mention). (A kludge is putting a 'sleep' somewhere to allow for propagation.)

So instead what you can do is have "_acme-challenge.svc1.example.com" be a CNAME that points to (say) "_acme-challenge.svc1.dnsauth.example.com". This sub-domain is not anycast, and may actually be a single machine that is used solely for this purpose.

The LE/ACME server goes to your main domain, finds a CNAME, and follows that to the real record and verification is achieved:

* https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...

* https://dan.langille.org/2019/02/01/acme-domain-alias-mode/

* https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...

The CNAME has to be set up initially, but can be left lying around otherwise.

This is how $WORK deals with getting LE certs for internal domains: we create a CNAME record (but no A records) for the internal hostname in our external DNS that point to our "dnsauth" domain which gets updating by internal clients via an API.

[e12e]

Oh, this is an interesting trick... I think I'll need to investigate further.

Do you use a custom acme/dns updater for automatic renewals?

[ed: ie - if I understand correctly, I could point: _acme-challenge.example.com via CNAME to auth.other.example.net - but then I'd like a command to check renew my example.com certs - and it would ideally use an api/dns update to manipulate the auth.other.example.net TXT (or CNAME to something like a15ce5b2-f170-4c91-97bf-09a5764a88f6.auth.acme-dns.io) record when I ask for a check/update of '*.example.com' certificate.

As far as I'm aware, most tooling assumes that you can/will (programmatically or manually) update the _acme-challenge.example.com record directly when issuing/updating an example.com certificate?]

[throw0101a]

The way it works for us:

First we use standard LE/ACME clients: either certbot or dehydrated. They ask for something like svc1.int.example.com ($DOMAIN).

In the hook script(s)† we manipulate the $DOMAIN string to put it into dnsauth.example.com ($AUTH_ZONE) sub-domain and send that new string to the DNS server that handles the dnsauth zone (and only that).

Before all of this we would have set up, in our public-external DNS, a CNAME record to point svc1.int to svc1.int.dnsauth.

The ACME client only thinks about $DOMAIN and the cert-issuing LE server only thinks about $DOMAIN. But the "in between" does not: by doing text manipulation (expr(1) is handy in shell scripts), and DNS redirects, the "in between" uses not-$DOMAIN for verification, but rather TXT records in $AUTH_ZONE.

All with standard ACME clients and some jiggery pokery.

We ended up creating some custom scripts called via SSH, but there are (now) DNS servers written specifically to handle REST API calls [0] and one can use lexicon [1] for just about any commercial DNS service.

[0] https://github.com/joohoi/acme-dns

[1] https://github.com/AnalogJ/lexicon

† dehydrated has deploy_challenge() and clean_challenge() functions in its example hook script. I'm sure most ACME clients have something similar.

[e12e]

Ok, thank you for the details and managing expectations. This still seems to warrant some experimentation.

In my case I'm mostly interested in delegating a domain/sub-domain somewhere I can easily update (be that run my own dns, host it somewhere with an api) - while having my main domains on a more boring/static dns infrastructure - yet still easily get certs for things like imap.example.com - which would not run a web server. And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.

[throw0101a]

> yet still easily get certs for things like imap.example.com - which would not run a web server.

Well, depending on the OS, you could start up a web server during the LE verification process and then bring it down once that's done. You'd only have to run in on port 80 for probably less than a minute.

But yes, you could this mechanism to have "_acme-challenge.imap.example.com" (which is what the ACME protocol uses) be a CNAME to point to something.auth.example.com that is more dynamic. Or even a completely different domain like foo.bar.example.ORG.

In your example.com zone file you'd put NS and A(AAA) records to point to the DNS server that handles the queries for the auth sub-domain.

> And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.

It's easier to run the ACME client on the host in question, and I'm not sure what it gains you to have it run somewhere else. That being said, there are ACME clients with a bit of a focus on being run 'remotely' from where the certs actually live:

* https://github.com/srvrco/getssl

This is probably for shared-hosting scenarios where cron is not accessible.

IMHO though, if you have access to the CLI on the host running the TLS service, it's best to run things there.

[e12e]

Re: your latest point - typically I'd like imap/smptd to run in separate static containers/vms with read access to the cert, but not write (and a volume or db to write emails to etc).

In general I'd prefer the certs be something the services get via configuration mgmnt - while the cert service can run via cron and make sure certs are valid an present.

In particular, I don't want my smtpd server to have write access to my dns, if I can help it.

[sleevi]

Where have you seen Let’s Encrypt using Google’s servers?

CAs are required to run full recursive resolvers, up to the root, and can’t just point at someone else’s DNS infrastructure. Which, if you think about it, is what you want: you don’t want the CA just trusting someone else is being honest, you want to go to the authoritative source.

[mercora]

i had this issue but i just set the time to wait for propagation high enough to be somewhat certain and had no further issues since (its 10m i think). it does not really matter to me how long it takes as its an automated process... should be finished before expiration though ^^

having multiple nameservers is pretty standard and often mandatory requirement set by the NICs. Its just that my secondary is sometimes not fast enough to transfer the zone after a change notification which triggers this issue.

Also, retrying after an authoritative nameserver said there is defacto no record seems pretty wrong to me... i doubt they use google DNS or anything really. in order to avoid caching issues they very likely resolve names recursively without (the usual) caching

[radiowave]

In which case it may be advisable to delegate the acme-challenge record to a different DNS provider, if doing so allows you to sidestep the anycast issues.