[LeonM]

This one bit me today and abruptly ended my day at the beach.

The certificate reseller advised my customer that it was okay to include the cross-signing cert in the chain, because browsers will automatically ignore it once it expires, and use the Comodo CA root instead.

And that was true for browsers I guess. But my customer also has about 100 machines in the field that use cURL to access their HTTPS API endpoint. cURL will throw an error if one of the certs in the chain has expired (may be dependent on the order, don't know).

Anyway, 100 machines went down and I had a stressed out customer on the phone.

[lucaspiller]

HTTP clients in programming languages are not as smart as web browsers when it comes to verifying SSL certificate chains. For example, if the chain presented by the server is missing intermediate certificates, modern web browsers are able to fetch those intermediate certificates without issue. Most HTTP clients do not do that though, and instead will throw a cryptic error, something along the lines of "unable to get local issuer certificate". This is known as a 'incomplete chain' error.

Earlier this year I added SSL verification to a website uptime monitoring service I run (https://www.watchsumo.com/docs/ssl-tls-monitoring) and it wasn't anywhere near as simple as I thought it would be. There's so many edge cases regarding verification, and languages usually don't expose the full errors in exceptions, then you have errors like this which only affect a subset of clients.

[tass]

Hi Luca, Just tried this out. I added a URL which has an expired root cert, but it passed your test.

Let me know if I can help with more info.

[0x0]

Sounds like a good test case for exercising those otherwise useless "million dollar insurances" that some certificate vendors flash in their sales materials?

[donmcronald]

Have you ever read the terms? I don't know if they even publish them anymore, but I read one many years ago. TLDR;

1. The CA must misissue a cert.

2. The misissued cert is used by a malicious party to impersonate you.

3. Every user (your users) must prove their damages and claim individually.

4. There might have been a low maximum, per-user claim, but I can't remember.

I'd be amazed if there's a single person on the internet who's been paid out by that warranty.

[tialaramex]

Yes, it's useless insurance. The interesting thing is that useless insurance is illegal to sell in lots of places† - to consumers, but here the insurance was sold to the root CAs which are huge corporations so they don't care that it's useless because they only bought it as a PR exercise.

†This is the root of a huge scandal in the UK that resulted in banks refunding people years of fees for a product called PPI which they should never have been sold. As a secondary effect this resulted in annoying spam from firms who'd help you claim your money back. By the end I almost felt sad I hadn't fallen for the original scam, because I was being inconvenienced by all the spam but (since I hadn't lost anything) not getting a pile of cash as a refund.

[nyuszika7h]

I had this problem with mediaarea.net. Actually, cURL and openssl s_client didn’t complain, but wget and APT failed because a certificate in one of the certification paths had expired. Had to contact them to fix it.

[mehrdadn]

Is that a cURL bug?

[mshade]

It seems only to be older versions of curl or curl with openssl <= 1.1.1. My macbook's curl fails, but my arch linux box's curl works fine.

[jonathanoliver]

When I started getting reports from customers on my company's website about issues, I was baffled for a few minutes because all my tests with OpenSSL and cURL, etc. were passing...on my Arch Linux install. Then I switched over to my MacBook and ran the official /usr/bin/curl version (instead of my brew install curl version) and I understood what was happening. Gotta love when this happens.

[kbenson]

My guess is openssl, since we experienced this problem with a lot of our internal services and our monitoring, both of which make heavy use of Perl and LWP::UserAgent, which build on OpenSSL. CentOS 7 boxes had problems (easily shown through the lwp-request util, which can often be used like curl's CLI tool), but not on CentOS 8.