[chipdart]

I loved the article. Insightful, and packed with real world applications. What a gem.

I have a side-question pertaining to cost-cutting with Kubernetes. I've been musing over the idea of setting up Kubernetes clusters similar to these ones but mixing on-premises nodes with nodes from the cloud provider. The setup would be something like:

- vCPUs for bursty workloads,

- bare metal nodes for the performance-oriented workloads required as base-loads,

- on-premises nodes for spiky performance-oriented workloads, and dirt-cheap on-demand scaling.

What I believe will be the primary unknown is egress costs.

Has anyone ever toyed around with the idea?

[mhuffman]

For dedicated they say this:

>All root servers have a dedicated 1 GBit uplink by default and with it unlimited traffic.

>Inclusive monthly traffic for servers with 10G uplink is 20TB. There is no bandwidth limitation. We will charge € 1/TB for overusage.

So it sounds like it depends. I have used them for (I'm guessing) 20 years and have never had a network problem with them or a surprise charge. Of course I mostly worked in the low double digit terabytes. But have had servers with them that handled millions of requests per day with zero problems.

[lyu07282]

20TB egress on AWS runs you almost $2,000 btw. one of the biggest benefits of Hetzner

[pdpi]

1 / 8 * 3600 * 24 * 30 = 324000 so that 1GBit/s server could conceivably get 324TB of traffic per month "for free". It obviously won't, but even a tenth of data is more than the data included with the 10G link.

[jorams]

They do have a fair use policy on the 1GBit uplink. I know of one report[1] of someone using over 250TB per month getting an email telling them to reduce their traffic usage.

The 10GBit uplink is something you need to explicitly request, and presumably it is more limited because if you go through the trouble of requesting it, you likely intend to saturate it fairly consistently, and that server's traffic usage is much more likely to be an outlier.

[1]: https://lowendtalk.com/discussion/180504/hetzner-traffic-use...

[chipdart]

> We will charge € 1/TB for overusage.

It sounds like a good tradeoff. The monthly cost of a small vCPU is equivalent to a few TB of bandwidth.

[adamcharnock]

We've toyed around with this idea for clients that do some data-heavy data-science work. Certainly I could see that running an on-premise Minio cluster could be very useful for providing fast access to data within the office.

Of course you could always move the data-science compute workloads to the cluster, but my gut says that bringing the data closer to the people that need it would be the ideal.

[threeseed]

> Has anyone ever toyed around with the idea?

Sidero Omni have done this: https://omni.siderolabs.com

They run a Wireguard network between the nodes so you can have a mix of on-premise and cloud within one cluster. Works really well but unfortunately is a commercial product with a pricing model that is a little inflexible.

But at least it shows it's technically possible so maybe open source options exist.

[SOLAR_FIELDS]

You could make a mesh with something like Netmaker to achieve similar using FOSS. Note I haven’t used Netmaker in years but I was able to achieve this in some of their earlier releases. I found it to be a bit buggy and unstable at the time due to it being such young software but it may have matured enough now that it could work in an enterprise grade setup.

The sibling comments recommendation, Nebula, does something similar with a slightly different approach.

[chipdart]

> They run a Wireguard network between the nodes so you can have a mix of on-premise and cloud within one cluster.

Interesting.

A quick search shows that some people already toyed with the idea of rolling out something similar.

https://github.com/ivanmorenoj/k8s-wireguard

[nullify88]

I believe the Cilium CNI has this functionality built in. Other CNIs may do also.

[sneak]

Slack’s Nebula does something similar, and it is open source.

[oblio]

I'm a bit sad the aggressive comment by the new account was deleted :-(

The comment was making fun of the wishful thinking and the realities of networking.

It was a funny comment :-(

[bdcravens]

Enable "showdead" on your profile and you can see it.

[rad_gruchalski]

It wasn’t funny. I can still see it. The answer was vpn. If you want to go fancy you can do istio with vms.

[ffsm8]

And if you wanna be lazy, there is a tailscale integration to run the cluster communication over it.

https://tailscale.com/kb/1236/kubernetes-operator

They've even improved it, so you can now actually resolve the services etc via the tailnet dns

https://tailscale.com/learn/managing-access-to-kubernetes-wi...

I haven't tried that second part though, only read about it.

[rad_gruchalski]

Okay, vpn it is.

[ffsm8]

I just wanted to provide the link in case someone was interested, I know you already mentioned it ｡ ◕ ‿ ◕ ｡

(Setting up a k8s cluster over software VPN was kinda annoying the last time I tried it manually, but super easy with the tailscale integration)

[juiyhtybr]

yes, like i said, throw an overlay on that motherfucker and ignore the fact that when a customer request enters the network it does so at the cloud provider, then is proxied off to the final destination, possibly with multiple hops along the way.

you can't just slap an overlay on and expect everything to work in a reliable and performant manner. yes, it will work for your initial tests, but then shit gets real when you find that the route from datacenter a to datacenter b is asymmetric and/or shifts between providers, altering site to site performance on a regular basis.

the concept of bursting into on-prem is the most offensive bit about the original comment. when your site traffic is at its highest, you're going to add an extra network hop and proxy into the mix with a subset of your traffic getting shipped off to another datacenter over internet quality links.

[threeseed]

a) Not every Kubernetes cluster is customer facing.

b) You should be architecting your platform to accomodate these very common networking scenarios i.e. having edge caching. Because slow backends can be caused by a range of non-networking issues as well.

c) Many cloud providers (even large ones like AWS) are hosted in or have special peering relationships with third party DCs e.g. [1]. So there are no "internet quality links" if you host your equipment in one of the major DCs.

[1] https://www.equinix.com.au/partners/aws

[chipdart]

> yes, like i said, (...)

I'm sorry, you said absolutely nothing. You just sounded like you were confused and for a moment thought you were posting on 4chan.

[rad_gruchalski]

Nobody said „do it guerilla-style”. Put some thought into it.