[throwawayboise]

I feel like I must be stupid because I've tried several times to set up k8s on in-house hardware and it has never worked. I've tried several different "recipes" and the networking never works right. IDK if it's because things are changing so fast that the popular "recipes" are already outdated, or if they are being published without actually testing them, but it's left a bad taste in my mouth.

I'm sure the big cloud providers make it easy for end users to use, but that doesn't help me.

[Fiahil]

You can start with K3s: https://k3s.io/

The networking part is always the most challenging. Everything between your router and your kubernetes cluster should still be routed and firewalled manually. However, if you can live with your home router firewall and a simple port mapping to your machines/cluster, then routing the traffic and setting up the cluster should be relatively painless.

[alanwreath]

This^^ I started out with kubespray because I was familiar with Ansible (I even contributed a very small bug fix), BUT k3s is just so awesome and out of your way. It’s not only easy to install but easy to remove.

[Okkef]

Don't run your own cluster, unless you do it to learn. If you're in the cloud, just use a managed instance and focus on building software.

[foobiekr]

How do people who do this debug issues in the k8s layer?

Or is this just the Jenga Model of Modern Software Development in action?

[GauntletWizard]

There's definitely a lot of Jenga going on here, but on the other hand: Kubernetes, when set up, has some very simple constraints that work. You don't often need to touch things at that layer; They become invisible, and when you're dealing with a cloud provider's hosted K8s, you don't get to touch them directly anyway.

K8s was a lot more simple earlier on. It's actually dying from adoption: There's a ton of dumb ideas bolted on top, that have become "standard" and "supported" because of demands from bad customers. The core is very clean, though, and you rarely need to interact with that.

[remram]

How does Ably deal with issues in the EC2 layer?

[foobiekr]

That's one of those great questions about AWS. We actually have had to contact AWS on multiple occasions about EC2 layer issues, and each time I was thankful that a VM construct is very simple, comparatively, to reason about.

[mycall]

Little known but latest VMware Fusion and VMware Workstation comes with kubernetes OOTB (vctl, kind). It has never been easier to start up a cluster.

[ofrzeta]

I don't know what specific problems you had with networking but I found using Kubespray an easy way to setup a cluster on different clouds.