[jrockway]

I doubt it. AWS's certs are just another three-quarters baked AWS feature. They did the best they could with the resources they had.

At my last job we had a fun and exciting outage when AWS simply didn't auto-renew our certificate. We were given no warning that anything was broken, and it apparently began the internal renewal process at the exact instant the cert expired (rather than 30 days in advance as is common with ACME-based renewal). Ultimately the root cause was that some DNS record in Route 53 went missing, and that silently prevents certificate renewal.

We switched TLS termination from the load balancer to Envoy + cert-manager and the results were much better. You also get HTTP/2 out of the deal. We also wrote a thing that fetches every https host and makes sure the certificate works, and fed the expiration times in prometheus to actually be alerted when rotation is broken. Both are features Amazon should support out of the box for the $20/month + $$/gigabyte you pay them for a TLS-terminating load balancer. Both are features Amazon says "you'll pay us anyway" to, and they're right.

[ti_ranger]

> it apparently began the internal renewal process at the exact instant the cert expired (rather than 30 days in advance as is common with ACME-based renewal).

Was this some time ago?

The FAQ for ACM (https://aws.amazon.com/certificate-manager/faqs/ ) says:

> Q: When does ACM renew certificates? > > ACM begins the renewal process up to 60 days prior to the certificate’s expiration date. The validity period for ACM certificates is currently 13 months. Refer to the ACM User Guide for more information about managed renewal.

> We switched TLS termination from the load balancer to Envoy + cert-manager and the results were much better. You also get HTTP/2 out of the deal. We also wrote a thing that fetches every https host and makes sure the certificate works, and fed the expiration times in prometheus to actually be alerted when rotation is broken. Both are features Amazon should support out of the box for the $20/month + $$/gigabyte you pay them for a TLS-terminating load balancer.

You're implying that AWS doesn't support HTTP/2 on any load-balancers they offer, but ALB has supported HTTP/2 since launch ( https://aws.amazon.com/blogs/aws/new-aws-application-load-ba... ) 3 years ago.

I don't see any current load-balancer priced at $20/month (ALB, NLB and Classic ELB are all ~ $8/month), so I can't guess which one you were using here ...

[jrockway]

I have no memory of when this was but it was on the order of 9 months to a year ago.

"up to 60 days before" includes "five minutes after". What it excludes is the renewal starting 61 days before the cert expires, and, as documented, it sure didn't do that.

Stuff went wrong and we had no observability. That is the AWS way.

[sciurus]

Not to be mean, but you definitely had observability into the expiration date of your certificate. You just weren't monitoring it yet. What you are doing now with Prometheus sounds good.

[toast0]

If you need to figure out for yourself what to monitor about the service, including things AWS says it handles, it brings into question the value of the service.

[tybit]

Interestingly while ALBs support serving HTTP2 client requests, they only proxy them to back ends using HTTP1.1. This breaks some use cases like gRPC unfortunately.

https://serverfault.com/questions/836568/terminate-http-2-on...