[jeffbee]

Quick reminder from your friendly local SRE: never ever issue certificates that expire on weekends. Make certs expire in the middle of the afternoon on a business day wherever your operators live and work. The cert in question expires at May 30 10:48:38 2020 GMT, which smells suspiciously like a fixed time after the cert was generated, rather than at a well-chosen point in time.

[lmilcin]

All my applications use a component that watches certs configured (everything in cert and trust store) and returns warning in telemetry from the application if any of the certificates is less than a week from expiration. This is checked periodically while the application runs.

This not only makes sure we don't miss expiration but also ensures we don't forget to configure any of the application.

We had a situation when the cert was replaced but the file was placed in incorrect path and was not actually used by the app. Having the app report on what is actually being in use is the best way to prevent this from ever happening.

[kl4m]

Good old "cert replaced but apache/nginx failed to reload" has bitten me more than once...

[Sebb767]

Me too! Especially with the short expiration times of LetsEncrypt. But I really don't want to put `nginx -s reload` in the Cron, in case I'm tinkering with the configs and they're suddenly live (which only really happens at staging or at home of course, but still).

[emilburzo]

You can use `nginx -t && nginx -s reload` for that.

It will first check the configs/paths, and only then, if successful, signal nginx to reload.

[Sebb767]

That's what I usually do. My problem is that I might be adding a location and nginx reloads between that and adding access restrictions (i.e. because I took a break to google).

[amaccuish]

Certbot has deploy hooks which is where I'd put the nginx reload statement. The hooks are run automatically when a new certificate is issued.

[Sebb767]

Oh, that's a great idea! Thanks :)

[colechristensen]

I've used this https://manpages.debian.org/testing/nagios-plugins-contrib/c...

After one scrambling emergency with a cert expiring in the middle of the day, a constant check with warnings and alerts a couple of weeks before expiry made a matter of defensive organization into something trivial.

[BurningFrog]

There is just no substitute for Reality!

[btgeekboy]

If you get to the point where the exact expiration date on the certificate matters, you've already lost the game.

[colechristensen]

Engineering for failure is important, you should always set yourself up so that you have several lines of defence which can fail. Some lines of defence to make failing "impossible" others to make a fail softer, even when you think failing is impossible.

[DavidSJ]

Defense in depth.

[techslave]

it’s more like blue m&m’s than an actual requirement

[alasdair_]

>it’s more like blue m&m’s than an actual requirement

Did you mean Van Halen's famous "WARNING: ABSOLUTELY NO BROWN M&Ms" clause?

https://www.snopes.com/fact-check/brown-out/

[dialamac]

Probably. It’s an honest mistake. For Gen Xers and early millennials blue m&ms are memorable for being added in the 90s with much ludicrous publicity.

[encoderer]

Great tip. Did you notice that cert in this case was issued 20 years ago? It’s crazy to me that it was still being used to sign certs as recently as last week (according to twitter)

[jeffbee]

Of course, but that doesn't really excuse them. My first experience with middle-of-Sunday-night SSL certificate expiration was in December 1998, and it was already a well-known doctrine by then. I'd expect a commercial certificate authority to have these kinds of things squared away.

[user5994461]

My experience with commercial CA is that they set the expiry exactly 1 year from creation. Doesn't matter if it's a week end or a holiday.

[sjtindell]

Generally that’s for Server certs, Roots and Intermediates will be multiple years from what I’ve seen. Roots in particular 10+ years.

[abiogenesis]

1 Year is a relatively recent thing. Previously you were able to buy server certs with a 5 year expiration.

[jis]

It's actually worse. The new root (good I believe until 2038) uses the same key as the now expired certificate. It has to or it would not be possible to validate the certificates that were issued. And this new one is a root certificate installed in browsers!

What "should" happen is that no certificate should be issued with an expiration date later than the issuing certificate. Then as the issuing certificate gets closer to expiration, a new one, with a new key pair, should be created and this new certificate should sign subordinate certificates.

[jis]

Sorry to reply to my own comment. But I want to clarify. Two certificates (at least) expired. The root named "AddTrust External CA Root" and a subordinate certificate with a subject of "USERTrust RSA Certification Authority." Both expired around the same time.

The "USERTrust RSA Certification Authority" certificate signed yet another layer of intermediate certificates.

The "USERTrust RSA Certification Authority" certificate was promoted to a self-signed certificate, now in the browser trust stores, using the same key pair as the original certificate that was signed by "AddTrust External CA Root." It has an expiration of 2038 (although that concept is a bit vague in a root certificate).

[josephcsible]

There's actually a third certificate for "USERTrust RSA Certification Authority", also using the same key pair, signed by a different root called "AAA Certificate Services". It looks like the intended replacement for the expiring one is this one, rather than the one where it's the root itself.

[dylz]

It is explicitly not a replacement, but some kind of legacy fallback that they don't want you to use, but exists for enterprise customers that absolutely can't get trust.

[josephcsible]

Are you sure? That's the path that InCommon has been providing me for new certificates since they switched away from the expiring one.

[cmbuckley]

Certificates issued with this CA will have been cross-signed by the newer root certificate, but our CA (Sectigo) was sending the old chain in issuing emails as late as April this year, despite the cross-signed root being available for a long time.

[imron]

> It’s crazy to me that it was still being used to sign certs as recently as last week (according to twitter)

It's likely because it was issued 20 years ago. People have been using it for 20 years and no-one realized it was about to stop working.

[adrianmonk]

> rather than at a well-chosen point in time

So, you're saying that "I'm not going to be working here anymore by then... hahahaha" isn't well-chosen?

[beamatronic]

Also make a Calendar placeholder (like a fake meeting), invite a lot of folks or a distribution list, and turn on an alert for 24 hours ahead.

[folmar]

For long-lived certificates they will outlive your calendar tech. The bit rot leave maybe the events fine, but anything fancier like notifications will fad away.

Source: BTDT 3 times in 7 years, and it was all with "Big Enterprise" grade products.

[jldugger]

> Make certs expire in the middle of the afternoon on a business day wherever your operators live and work.

If I can predict that 20 years into the future I wouldn't be in the SRE business.

[jeffbee]

I think it was foreseeable even in dark days of the year 2000 that this certificate expired after business hours globally.

But your statement is really pointing out that nobody should be making long-lived certificates.

[solarengineer]

If you were to issue certs for short durations, and also maintain a calendar of cert expiry, those certs could be renewed in a timely manner.

In other scenarios where one would want to issue fresh certificates (receiving Ops control from other orgs, credentials refresh for what ever reason), one can still do so without waiting for the current certificates to expire.

[kjaftaedi]

"Middle of the afternoon" .. for who?

[hombre_fatal]

If you're going to ask a question about someone's comment, at least finish reading the sentence to make sure it's not immediately answered.

Don't be part of the death of internet discourse.

[kjaftaedi]

I read the comment several times.

I am just of the opinion that "make it expire in the afternoon" doesn't apply to root certificates that are used across the entire world (i.e. - the topic of discussion)