Hacker News new | ask | show | jobs
by kuhsaft 915 days ago
The node would be a virtual-kubelet. You can check out the virtual-kubelet GitHub repo for more info.

Interestingly, there are already multiple providers of virtual-kubelet. For example, Azure AKS has virtual nodes where pods are Azure Container Instances. There’s even a Nomad provider.

> So that’s what we do. When you create a cluster, we run K3s and the Virtual Kubelet on a single Fly Machine.

So probably a cluster per region. You could theoretically spin up multiple virtual-kubelets though and configure each one as a specific region.

> Because of kine, K3s can manage multiple servers, but also gracefully runs on a single server, without distributed state.

This would mean the control-plane would be on a single-server without high-availability? Although, I suppose there really isn’t any state stored since they are just proxying requests to the Fly Machine API. But still, if the machine went down your kubectl commands wouldn’t work.

1 comments

The diagram on https://virtual-kubelet.io/docs/architecture/ makes me wonder whether it's possible to have a k8s cluster where the nodes are all virtual kubelets backed by different cloud providers (and then perhaps schedule loads preferentially with selectors)
I think it’s completely possible. Though, you’ll have to manage your own control-plane.

Azure AKS and EKS provide virtual-kubelet functionality in some form, but AKS is an a managed control-plane where you can’t add nodes yourself and EKS only allows nodes in the same VPC.

Edit: It already is a thing. https://github.com/virtual-kubelet/tensile-kube

tensile-kube seems to be structured as a "k8s cluster of k8s clusters", with an upper kubemaster farming out resources to lower kubemasters (through virtual-node). I don't know if there's any particular reason to have that separation; possibly the lower kubemasters could be removed and you could just run a bunch of virtual-kubelets.
I think the biggest hurdle would be networking between the pods since they will be running on different cloud providers.
I've seen some people using wireguard for intra-cluster networking so that all their nodes can run pretty much anywhere.
Wouldn't the network cost be absurd in such case? Not only the pod-to-pod communication cost skyrocket, all the heartbeats, health checks, metrics, daemonsets pinging each other will probably end up costing more than the CPU and Memory