To me it sounds like the problem wasn't kubernetes, the problem was that you (your predecessor) rolled your own instead of going down the path of using something with a support number. Redhat obviously comes to mind first but there are countless options with enterprise support included.
Have you considered rebuilding/moving the containers onto something more "enterprisey"?
A 'support number' does not solve the technical issues with kubernetes, it just kicks the responsibility down the line so that someone else has to deal with it.
A "support number" gets you access to an expert. No offense to OP, but it doesn't sound like he's got a deep knowledge of the internals of Kubernetes. Which is fine, because it also sounds like that's not his main job, just something he's tasked with taking care of. That's literally why enterprise support exists. If we all had infinite time and brainpower to be experts in everything, we could just roll-our-own. But we don't. Which is why AWS exists, and IBM bought Redhat for billions of dollars.
Now that the myth of experts have been accurately described, let's say a few words about the reality.
If IBM wanted the experts, they would have hired or grown their own. What they wanted was, I guess, the contacts (actual and prospectives).
My experience with support contacts is that you often times have access only to someone who is not any more expert than you, and who care much less for your customers than you do. In several occasions it turned out the "expert" had been the one benefiting from the teaching from the in house "not supposed to be expert but knows more than the expert" guy (and yes also, and maybe especially, with "reputable" large companies like IBM or Oracle).
I can even remember of a particular instance when the expert had no access to his company internal documentation to get details about a specific error message we were hitting, and we had to find a pirate copy of some internal manual from some Russian website and hand it over to him.
It makes sense to have a service contract when you have really no knowledge at all on the domain, but as soon as it's related to your daily job then you will quickly realise that experts are mythical characters whom your contractor have no better access to than any other company, including your own.
>It makes sense to have a service contract when you have really no knowledge at all on the domain, but as soon as it's related to your daily job then you will quickly realise that experts are mythical characters whom your contractor have no better access to than any other company, including your own.
I would seriously question who you're doing business with. Anytime I have had a significant issue, the escalation path is to the guy who wrote the code. To imply that enterprise support is just a bunch of people who don't know more than the average guy off the street is ridiculous. Tier 1 support? Sure, but you don't stay there long if you're a clued user with a real issue.
That exemple was from a time I worked on a project for a big Telco, and support was (supposed to be) provided by a very large database company which database and file system we were using. I think the guy who wrote that code was very long gone.
I’m not trying to defend kubernetes complexity or saying it should be used for deployment of all server-side software, but I’d push back on the oft-repeated idea that kubernetes is just solving for scale. It’s a model of deployment that happens to make scaling easy, but the model has many advantages other than scale.
Resume-driven development is unfortunate; but a valid strategy when employers who run everything on one server that hasn't been updated since 2003 start listing 5 years of Kubernetes experience as a requirement in their job posting.
Have you considered rebuilding/moving the containers onto something more "enterprisey"?