[sergiosgc]

If you let a server lag in OS version, at some point in time you're going to hit this kind of problem. If not with Let's Encrypt, then with some other dependency. I know, I've been in the exact same spot. I just don't blame the dependency, and included server OS updates as part of a yearly maintenance cycle.

[jacquesm]

I find that really ridiculous. Not you, but the fact that an OS needs to be upgraded because of some application level stuff that has to do with a protocol that is being run on some other server.

That's the kind of dependency snowball that we should work hard to avoid, not accept as some kind of new normality.

Servers should be able to live for years without re-imaging.

[yebyen]

Is there a reason you can't just upgrade that one component on the server, why do you have to re-image it from scratch?

If you have external dependencies they are going to move around from time to time throughout their lifetimes, especially if they are beta. LetsEncrypt may not have signaled beta with v1, but I've been a cert-manager user for years in pre-1.0 and I've known that meant I might need to come up for air and read the docs for a specific upgrade instruction from one pre-1.0 minor version to another at any time.

Now cert-manager is 1.0+ and my expectations can change. It should remain backwards compatible until the next major version (hopefully for a while! And they will provide a migration path when that comes, with clear instructions and a fairly long sunset, godwilling)

But cert-manager depends on letsencrypt, and I depend on cert-manager, all of which depends on a protocol called acme, and this is the arrangement. We made this deal because it was going to turn out less complicated than managing the certificates by hand, and they made that deal because it was going to turn out better than rolling their own protocol from absolutely scratch, similarly. Eyes on the prize.

If you didn't want LetsEncrypt as a dependency there are other ways to connect cert-manager or another tool like it, including other acme providers... they all depend on the acme protocol, (or there might be some other protocol that you can use, with its own characteristics of change or stability, or roll your own) at some point you have to roll the dice and bet on something.

Occasionally these things happen. You suggest that servers should be able to go for years, (but they have allowed years for this transition! What more can be expected, realistically?)

[jacquesm]

> Is there a reason you can't just upgrade that one component on the server, why do you have to re-image it from scratch?

Yes, I did this now and I have it working. But it leaves things in a messed up state and I don't like that so I will go back to this in a short while and fix it properly.

What I still wonder about is why their warning email never reached me, that I really need to figure out because then at least I would have dealt with this under a lot less time pressure.

> If you didn't want LetsEncrypt as a dependency there are other ways to connect cert-manager or another tool like it, including other acme providers...

There are some very good suggestions in this thread, I will probably adopt one of them.

> You suggest that servers should be able to go for years, (but they have allowed years for this!)

And somehow I missed that memo. Even so, I am still not convinced of the necessity, it is possible that it exists but I have yet to see a valid reason for shutting down the old protocol for new registrations like this. There also seems to be some confusion with people saying it should have worked for the same account, which I can prove did not work.

[yebyen]

> But it leaves things in a messed up state and I don't like that so I will go back to this in a short while and fix it properly.

You say this with confidence, I wish my own situation provided me with the confidence to say this and mean it. We do not have reproducible systems and depend in many ways wholly on backup images of live production systems. Someone is going to say this makes my life simpler than yours by some twisted math, but I have a doubt about that myself.

We are still talking about migrating from Amazon Linux v1 to Amazon Linux v2, and with a recent announcement from AWS, the pressure is off! We'll be able to continue talking about this transition for a good long time to come. Again, mixed blessing, is it better to have an operating system that can crawl along on life support? For those that can't upgrade, sure, it is better to get security maintenance than to have zombie servers which are not upgradeable, but who is to say what opportunity costs will arise because we are not on a formally supported leading-edge version of the platform.

[jacquesm]

Agreed, reproducible systems are an absolute must and it is a shame that we are still not even close to having a solid foundation under all this mission critical stuff we build.

It feels like we are building these huge castles on quicksand.

At the same time I think the whole 'treat your servers like cattle, not like pets' is exactly because we don't know how to do this properly. It is the cloud equivalent to hitting ctrl-alt-delete to solve issues.