[jrockway]

Something that's nice about Let's Encrypt is that it forces you to change something every few months. After the first couple months, you'll probably get your issues worked out. If you just change certs every few years, then every few years you have some sort of disaster because of the "well we fixed it, we don't have to worry for two years" effect.

A broader lesson is the importance of "trying out" rare events, even before that rare event actually happens. If depends on a service with a certain SLA, it's pretty dangerous when that service has 100% uptime. You never get to see what happens when it does go down, which it did promise you it will. Some people track their error budget, and at the end of the accounting period, break their service in accordance with the SLA. Then you get to see what happens when it does go down. (Ref: https://queue.acm.org/detail.cfm?id=2371516)

[tialaramex]

Although, speaking of Let's Encrypt, there will be a series of disruptive events over the next 18 months or so.

* Soon (although when exactly I'm not sure because it has been delayed at least once) the Let's Encrypt systems will tell compliant ACME clients that the "correct" intermediate is Let's Encrypt's ISRG-signed X3 intermediate. This is a different certificate for the same X3 private key you're used to but not signed by the same trust root. If you use a correct client and have done things properly, this may cut off TLS clients for your systems that don't trust ISRG (the charity which runs Let's Encrypt). Six year old Android phones, the Windows XP system you know should have been retired, a VoIP desk phone running out-of-date firmware, stuff like that.

* In March 2021 the X3 Intermediate expires. If your certificate software was not compliant with ACME, or you manually overrode it to use the old certificates to avoid the problem in the previous item, things break now. More things, and worse. Although...

* Maybe before March 2021 the Let's Encrypt systems stop issuing from those soon-to-be-obsolete Intermediates and use newer ones instead perhaps named Y3 and Y4. In this case if you've jury rigged things (in an ACME non-compliant way) to keep using the old X3 intermediate that'll break suddenly after your renewal. Common web browsers may not trust the nonsense you're emitting, exactly which browsers break may vary depending on exactly what stupid things you did, but chances are you haven't tested and don't know. If you are using a compliant client then modern browsers are all fine, but archaic stuff breaks suddenly.

* In September 2021 the DST Root X3 root expires. If you have somehow clung on to trust via this root, whether through your own effort or via trust path discovery code inside client systems, that goes away instantly. Any systems that don't trust ISRG will refuse to trust your certificates, no matter how often you re-issue them and reconfigure things, those clients themselves need updating urgently and you probably have no way to do that. Oops.

[Denvercoder9]

That sounds like just one maybe-disruptive event that manifests itself differently if you keep working around it instead of dealing with it properly. If you need to deal with it at all - I suspect most systems that still need to connect to the internet trust the ISRG root nowadays.

[EB66]

> I suspect most systems that still need to connect to the internet trust the ISRG root nowadays.

There are tons of systems that do not -- particularly in the enterprise. I manage web servers for a mission-critical healthcare-related SaaS. We occasionally encounter TLS issues even with Globalsign root certificates -- far more distributed than ISRG.

We ended up switching to DigiCert last year and it helped reduce the number of TLS-related failures reported to us.

We could never switch to Let's Encrypt / ISRG for that reason. Even if ISRG has 95% distribution of their root certificate, that's not good enough for mission-critical enterprise.

I'm not at all surprised that Heroku had to roll back their TLS certificate back to DigiCert -- DigiCert is what you want if need compatibility with the highest number of clients.

[floatingatoll]

This inspired me to look into my system's trusted roots. Here's the root CA expirations coming up in the next 18 months. The last one on this list really hits home, as anyone who did TLS back in the early 00's may remember.

2020-09-12 - DST Root CA X4

2021-03-17 - QuoVadis Root Certification Authority

2021-04-06 - Sonera Class X2

2021-09-30 - DST Root CA X3

2021-11-09 - Admin-Root-CA

2021-12-15 - Belgium Root CA2

2021-12-15 - GlobalSign

[tialaramex]

Admin-Root-CA shows us how far we've come, I think today that even if Mozilla's root programme didn't forbid them people would guess that ultra-vague names aren't a good idea.

For reference that is the Swiss government's root and it isn't trusted by Mozilla so as a consequence it's unlikely that any systems you have facing ordinary web browsers depend on this root to be trusted.

It's also funny to go back and look at Mozilla's trust decision (it's before I was engaged in looking at this on a day-to-day basis) and see that the terrible naming was decisive while the practice of just basically trusting a Swiss government employee to issue whatever they want was considered only "problematic" and not necessarily a showstopper.

Of course because Mozilla doesn't trust this root, it does not see itself as having any oversight role for the root. So if you use MacOS, or Windows, to do anything other than run Firefox, you're reliant on their teams to verify that this root is well run. Maybe they're doing a great job? I guess you'd only ever find out the hard way because they operate entirely behind closed doors.

[tinus_hn]

Certificate Transparency logs would provide the answer to this.

[mjw1007]

I think "Six year old Android phones" is optimistic. As I understand it the ISRG root was added in Android 7.1, and 7.0 was released in August 2016.

So it might be more like "Three year old Android phones", given the lag between upstream releases and adoption.

[fomine3]

https://news.ycombinator.com/item?id=23496332

They postponed but anyway they planned to drop old Android support in 2020 but I doubt it's possible in near future.

[tinus_hn]

Security means you don’t support

> Six year old Android phones, the Windows XP system you know should have been retired, a VoIP desk phone running out-of-date firmware, stuff like that.

If these can’t connect that is a feature, not a bug.

[tgsovlerkhgsel]

The downside is that due to a lack of serious competition, Let's Encrypt seems like an obvious choice, and thus it can be tempting to hardcode it.

I have a homebrew Internet-of-shit device that I know has LE hardcoded. I'll have to take it of the wall and reflash if I switch to a new CA (or potentially when some of the changes described by tialaramex happen - I think I hardcoded the new root but I'm not 100% sure).

[manquer]

The acme protocol is well defined , and code is open source you could always implement your own service.

Let’s encrypt only real hold is their root certificate is now in many trust stores , if you control both sides self signed certificates are perfectly fine you don’t need a CA at all

[jrockway]

I think he's talking about the temptation to set up a pin to their root. That can break just as easily as any other pin, and of course you won't be prepared.